Linux version of PMS with statically linked ffmpeg/mencoder

[happy.neko]

Tests with clean install of Centos 6 are successful. Transcoding works fine. There were some problems with stand-alone libmediainfo, I fixed them in latest version of the script.

[langerak]

I'll give it a try tomorrow on my ubuntu 11.04 server and see what it results in.

For now, nice work!

[langerak]

I've given the script a run on my server (Ubuntu 11.04 64-bit) , but to no avail for now. When compiling zlib at first it goes wrong:

Code: Select all

root@ubuntu:~/contrib# ./build-pms-linux.sh


--------------------------------------------------------------------------------------
Building zlib
--------------------------------------------------------------------------------------

--2011-09-29 08:06:03--  http://zlib.net/zlib-1.2.5.tar.gz
Resolving zlib.net... 69.73.181.135
Connecting to zlib.net|69.73.181.135|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 544640 (532K) [application/x-gzip]
Saving to: `zlib-1.2.5.tar.gz'

100%[=====================================================================================================================================================================>] 544,640      380K/s   in 1.4s

2011-09-29 08:06:06 (380 KB/s) - `zlib-1.2.5.tar.gz' saved [544640/544640]

Checking for gcc...
Checking for shared library support...
Tested cc -w -c -march=pentium4 -mtune=i686 -I/root/contrib/target/include ztest9371.c
ztest9371.c:1:0: error: CPU you selected does not support x86-64 instruction set
Tested cc -shared -march=pentium4 -mtune=i686 -I/root/contrib/target/include -o ztest9371.so ztest9371.o
cc: ztest9371.o: No such file or directory
cc: no input files
No shared library support; try without defining CC and CFLAGS
Building static library libz.a version 1.2.5 with cc.
Checking for off64_t... No.
Checking for fseeko... No.
Checking for unistd.h... No.
Checking whether to use vs[n]printf() or s[n]printf()... using s[n]printf().
Checking for snprintf() in stdio.h... No.
  WARNING: snprintf() not found, falling back to sprintf(). zlib
  can build but will be open to possible buffer-overflow security
  vulnerabilities.
Checking for return value of sprintf()... No.
  WARNING: apparently sprintf() does not return a value. zlib
  can build but will be open to possible string-format security
  vulnerabilities.
cc -march=pentium4 -mtune=i686 -I/root/contrib/target/include -DNO_FSEEKO -DNO_snprintf -DHAS_sprintf_void   -c -o example.o example.c
cc -march=pentium4 -mtune=i686 -I/root/contrib/target/include -DNO_FSEEKO -DNO_snprintf -DHAS_sprintf_void   -c -o adler32.o adler32.c
cc -march=pentium4 -mtune=i686 -I/root/contrib/target/include -DNO_FSEEKO -DNO_snprintf -DHAS_sprintf_void   -c -o compress.o compress.c
cc -march=pentium4 -mtune=i686 -I/root/contrib/target/include -DNO_FSEEKO -DNO_snprintf -DHAS_sprintf_void   -c -o crc32.o crc32.c
cc -march=pentium4 -mtune=i686 -I/root/contrib/target/include -DNO_FSEEKO -DNO_snprintf -DHAS_sprintf_void   -c -o deflate.o deflate.c
cc -march=pentium4 -mtune=i686 -I/root/contrib/target/include -DNO_FSEEKO -DNO_snprintf -DHAS_sprintf_void   -c -o gzclose.o gzclose.c
cc -march=pentium4 -mtune=i686 -I/root/contrib/target/include -DNO_FSEEKO -DNO_snprintf -DHAS_sprintf_void   -c -o gzlib.o gzlib.c
cc -march=pentium4 -mtune=i686 -I/root/contrib/target/include -DNO_FSEEKO -DNO_snprintf -DHAS_sprintf_void   -c -o gzread.o gzread.c
gzread.c:1:0: error: CPU you selected does not support x86-64 instruction set
make: *** [gzread.o] Error 1
make: *** Waiting for unfinished jobs....
compress.c:1:0: error: CPU you selected does not support x86-64 instruction setexample.c:1:0: error: CPU you selected does not support x86-64 instruction set

adler32.c:1:0: error: CPU you selected does not support x86-64 instruction setgzclose.c:1:0: error: CPU you selected does not support x86-64 instruction set

gzlib.c:1:0: error: CPU you selected does not support x86-64 instruction set
make: *** [compress.o] Error 1deflate.c:1:0: error: CPU you selected does not support x86-64 instruction set

make: *** [adler32.o] Error 1
make: *** [gzclose.o] Error 1
make: *** [deflate.o] Error 1
make: *** [gzlib.o] Error 1
make: *** [example.o] Error 1
crc32.c:1:0: error: CPU you selected does not support x86-64 instruction set
make: *** [crc32.o] Error 1
Fatal error occurred, aborting build.

This was tested using the archs like "generic", "i686" and "core2", altho it seems to grep something out which it fails at. I a sure that my CPU is 64-bit capable as it is a Xeon:

Code: Select all

root@ubuntu:~/contrib# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      : Intel(R) Xeon(TM) CPU 2.66GHz
stepping        : 4
cpu MHz         : 2659.728
cache size      : 2048 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 6
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts nopl pni dtes64 monitor ds_cpl vmx est cid cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips        : 5319.45
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      : Intel(R) Xeon(TM) CPU 2.66GHz
stepping        : 4
cpu MHz         : 2659.728
cache size      : 2048 KB
physical id     : 1
siblings        : 4
core id         : 0
cpu cores       : 2
apicid          : 4
initial apicid  : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 6
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts nopl pni dtes64 monitor ds_cpl vmx est cid cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips        : 5320.25
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      : Intel(R) Xeon(TM) CPU 2.66GHz
stepping        : 4
cpu MHz         : 2659.728
cache size      : 2048 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 2
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 6
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts nopl pni dtes64 monitor ds_cpl vmx est cid cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips        : 5320.23
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      : Intel(R) Xeon(TM) CPU 2.66GHz
stepping        : 4
cpu MHz         : 2659.728
cache size      : 2048 KB
physical id     : 1
siblings        : 4
core id         : 0
cpu cores       : 2
apicid          : 5
initial apicid  : 5
fpu             : yes
fpu_exception   : yes
cpuid level     : 6
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts nopl pni dtes64 monitor ds_cpl vmx est cid cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips        : 5320.23
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 4
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      : Intel(R) Xeon(TM) CPU 2.66GHz
stepping        : 4
cpu MHz         : 2659.728
cache size      : 2048 KB
physical id     : 0
siblings        : 4
core id         : 1
cpu cores       : 2
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 6
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts nopl pni dtes64 monitor ds_cpl vmx est cid cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips        : 5320.21
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 5
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      : Intel(R) Xeon(TM) CPU 2.66GHz
stepping        : 4
cpu MHz         : 2659.728
cache size      : 2048 KB
physical id     : 1
siblings        : 4
core id         : 1
cpu cores       : 2
apicid          : 6
initial apicid  : 6
fpu             : yes
fpu_exception   : yes
cpuid level     : 6
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts nopl pni dtes64 monitor ds_cpl vmx est cid cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips        : 5320.26
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 6
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      : Intel(R) Xeon(TM) CPU 2.66GHz
stepping        : 4
cpu MHz         : 2659.728
cache size      : 2048 KB
physical id     : 0
siblings        : 4
core id         : 1
cpu cores       : 2
apicid          : 3
initial apicid  : 3
fpu             : yes
fpu_exception   : yes
cpuid level     : 6
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts nopl pni dtes64 monitor ds_cpl vmx est cid cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips        : 5320.20
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 7
vendor_id       : GenuineIntel
cpu family      : 15
model           : 6
model name      : Intel(R) Xeon(TM) CPU 2.66GHz
stepping        : 4
cpu MHz         : 2659.728
cache size      : 2048 KB
physical id     : 1
siblings        : 4
core id         : 1
cpu cores       : 2
apicid          : 7
initial apicid  : 7
fpu             : yes
fpu_exception   : yes
cpuid level     : 6
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts nopl pni dtes64 monitor ds_cpl vmx est cid cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips        : 5320.23
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 48 bits virtual
power management:

Maybe you can substitute something from my CPU output .

When building the source in /src myself all goes ok, so I think your checks are too restrictive in this purpose or I am giving the wrong arch .

[langerak]

I've made a good progress just now with some modifications in your script.

As from line 126 you have the following lines:

Code: Select all

export CFLAGS="-march=pentium4 -mtune=$TARGET_ARCHITECTURE"
export LDFLAGS="-march=pentium4 -mtune=$TARGET_ARCHITECTURE"
export CXXFLAGS="-march=pentium4 -mtune=$TARGET_ARCHITECTURE"

You should replace that with the following:

Code: Select all

export CFLAGS="-march=native -mtune=$TARGET_ARCHITECTURE"
export LDFLAGS="-march=native -mtune=$TARGET_ARCHITECTURE"
export CXXFLAGS="-march=native -mtune=$TARGET_ARCHITECTURE"

The "native" flag lets the compiler choose the platform specific optimizations by itself and now the script runs. With this flag you let the compiler detect optimizations for 32 and 64-bit itself and works on both architectures.

On line 78 you have the following:

Code: Select all

TARGET_ARCHITECTURE="core2"

Turn that into:

Code: Select all

TARGET_ARCHITECTURE="generic"

That results in the same explanation used on the -march flag and makes it less independent and more workable on all X86(_64) platforms such as AMD / Intel.

I've also received an error about autoconf, the script does not check if this is installed, fixed by issuing:

Code: Select all

apt-get install autoconf automake

On line 1262 there is:

Code: Select all

$GPP -shared -static-libstdc++ -static-libgcc -o ./../libmediainfo.so ./*.o  -Wl,--whole-archive -L./ -lzen -Wl,--no-whole-archive

That results in an error stating that you want to create a shared library with static options, I managed to get the compile further by changing it into:

Code: Select all

$GPP -static-libstdc++ -static-libgcc -o ./../libmediainfo.so ./*.o  -Wl,--whole-archive -L./ -lzen -Wl,--no-whole-archive

But then it halts with:

Code: Select all

/usr/lib/x86_64-linux-gnu/gcc/x86_64-linux-gnu/4.5.2/../../../crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
./File_Ibi_Creation.o: In function `MediaInfoLib::File_Ibi_Creation::Finish()':
File_Ibi_Creation.cpp:(.text+0x55a): undefined reference to `compress2'
./File_Ibi.o: In function `MediaInfoLib::File_Ibi::CompressedIndex()':
File_Ibi.cpp:(.text+0x5a87): undefined reference to `uncompress'
./File_Mpeg4_Elements.o: In function `MediaInfoLib::File_Mpeg4::moov_cmov_cmvd_zlib()':
File_Mpeg4_Elements.cpp:(.text+0xd76d): undefined reference to `uncompress'
./File_Swf.o: In function `MediaInfoLib::File_Swf::Decompress()':
File_Swf.cpp:(.text+0xa35): undefined reference to `uncompress'
.//libzen.a(Thread.o): In function `ZenLib::Thread::Run()':
Thread.cpp:(.text+0x38e): undefined reference to `pthread_create'
collect2: ld returned 1 exit status
Fatal error occurred, aborting build.

Maybe I am missing some components? Can you check this out yourself?

Hope this helps you improve your script. I'm a Bash programmer as well and it is possible to let all dependencies install automatically, maybe this is something you want to look for .

[langerak]

And as an extra, if you let people install "sysstat":

Code: Select all

apt-get install sysstat

You can manage to find out how many core/threads the current machine has and adopt that in the amount of building threads in your script.

On my server this command resulted in the amount of cores:

Code: Select all

mpstat | grep "Linux" | awk '{print $6}' | tr -d \(

Results in:

Code: Select all

8

Via this method you always make use of all avaliable cores .

[happy.neko]

langerak, thanks for feedback. I think "-march=native" will compile highly optimized code for you CPU but not runnable on previous models. This script is intended to be run not by end-users but project maintainers to generate distributable binaries. So they have to be compatible with wide range of hardware platforms while being as optimized for speed as possible.

The idea of compiling in 64-bit environment didn't cross my mind. You can try "-m32" flag to compile 32-bit binaries and theoretically this should work with "-march=native -mtune=$TARGET_ARCHITECTURE" but I'm really clueless here. Can you test it?

Anyway I have updated my script with automake/autoconf checks and some fixes to make dcraw and flac statically linked.

Code: Select all

dcraw:                       ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.15, stripped
ffmpeg:                      ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.6.15, stripped
flac:                        ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.15, stripped
libmediainfo.so:             ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped
mencoder:                    ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.6.15, stripped
mplayer:                     ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.6.15, stripped

Now all generated tools should be ready for distribution. Can you please download (link) prebuild-linux-tools pack and test them on your system? I will look into your compilation errors and number of CPU cores autodetection later.

[happy.neko]

Good idea about amount of compilation threads. I think we can count the number of cores without extra tools with:

Code: Select all

cat /proc/cpuinfo | grep processor | wc -l

Script updated.

[langerak]

How should I test the tools? Just download the regular PMS and replace the files by this one?

[happy.neko]

Just drop them into pms-linux-1.40.0/linux folder and link or copy to /usr/local/bin.
You shoild get:

Code: Select all

> ffmpeg -version
ffmpeg version N-31460-g80e4fe4, Copyright (c) 2000-2011 the FFmpeg developers

> mencoder
MEncoder SVN-r34118-4.5.2 (C) 2000-2011 MPlayer Team

[langerak]

Ah, that's to test the functionality, I will test it tonight when I'm home .

The build process is good enough for the maintainers, but for me as a poweruser, I would like to build this stuff myself, maybe you want to create a userscript with the generic and native options applied to them . I never tend to download prebuild versions if I can build it myself and setting it up for my usage .

Have you looked into this as well:
I've also received an error about autoconf, the script does not check if this is installed, fixed by issuing:

Code: Select all

apt-get install autoconf automake

On line 1262 there is:

Code: Select all

$GPP -shared -static-libstdc++ -static-libgcc -o ./../libmediainfo.so ./*.o  -Wl,--whole-archive -L./ -lzen -Wl,--no-whole-archive

That results in an error stating that you want to create a shared library with static options, I managed to get the compile further by changing it into:

Code: Select all

$GPP -static-libstdc++ -static-libgcc -o ./../libmediainfo.so ./*.o  -Wl,--whole-archive -L./ -lzen -Wl,--no-whole-archive