PAR::Tutorial - Cross-Platform Packaging and Deployment with PAR

•

Q: "Help! I can’t run your program!"

•

A1: Install Perl & "perl −MCPAN −e'install(...)'"

How do we know which modules are needed?

•

New versions of CPAN modules may break "sshnuke.pl"

•

•

Possibly overwriting existing modules; not cross-platform at all

•

•

Impossible to debug; "perlcc" usually does not work anyway

•

Do what JAR (Java Archive) does for Perl

Aggregates modules, scripts and other files into a Zip file

•

Easy to generate, update and extract

•

Version consistency: solves forward-compatibility problems

•

Developed by community: "[email protected]"

•

•

Automatically scans perl script for dependencies

•

Bundles all necessary 3rd−party modules with it

•

Requires only core Perl to run on the target machine

•

PAR also comes with "pp", the Perl Packager:

•

PAR files are just Zip files with modules in it

•

Any Zip tools can generate them:

•

To load modules from PAR files:

•

This also works:

•

Use "par.pl" to run files inside a PAR archive:

•

Same thing, with the stand-alone "parl" or "parl.exe":

•

The PAR loader can prepend itself to a PAR file:

"−b" bundles non-core modules needed by "PAR.pm":

•

"−B" bundles core modules in addition to "−b":

•

Recursively scan dependencies with "scandeps.pl":

•

Scan an one-liner, list all involved files:

•

Combines scanning, zipping and loader-embedding:

•

Bundle additional modules:

•

Pack one-liners:

•

Generate PAR files instead of executables:

•

Command-line options are almost identical to "perlcc"’s

Also supports "gcc"−style long options:

•

Small initial overhead; no runtime overhead

•

Dependencies are POD-stripped before packing

•

Loads modules directly into memory on demand

•

Shared libraries (DLLs) are extracted with File::Temp

•

Works on Perl 5.6.0 or above

•

Tested on Win32 (VC++ and MinGW), FreeBSD, NetBSD, Linux, MacOSX, Cygwin, AIX, Solaris, HP-UX, Tru64...

•

A common question:

•

The obvious workaround:

•

On platforms that have "ln", there is a better solution:

•

Of course, there is no cross-platform binary format

•

Pure-perl PAR packages are cross-platform by default

However, XS modules are specific to Perl version and platform

•

Multiple versions of a XS module can co-exist in a PAR file

•

•

Now it works on both platforms:

•

Modules can reside in several directories:

•

Scripts are stored in one of the two locations:

•

Shared libraries may be architecture− or perl-version-specific:

•

PAR files may recursively contain other PAR files:

•

MANIFEST

Index of all files inside PAR

•

Can be parsed with "ExtUtils::Manifest"

•

•

Dependency, license, runtime options

•

Can be parsed with "YAML"

•

•

OpenPGP-signed digital signature

•

Can be parsed and verified with "Module::Signature"

•

This is not meant to be a flame

All three maintainers have contributed to PAR directly; I’m grateful

•

perlcc

"The code generated in this way is not guaranteed to work... Use for production purposes is strongly discouraged." (from perldoc perlcc)

•

Guaranteed to not work is more like it

•

•

Expensive: Need to pay for each upgrade

•

Non-portable: Only available for limited platforms

•

Proprietary: Cannot extend its features or fix bugs

•

Obfuscated: Vendor and black-hats can see your code, but you can’t

•

Inflexible: Does not work with existing Perl installations

•

The URL of "MANIFEST" inside "/home/autrijus/foo.par":

•

Open it in a Gecko browser (e.g. Netscape 6+) with Javascript enabled:

•

No needed to unzip anything; just click on files to view them

•

Static, machine-readable distribution metadata

Supported by "Module::Build", "ExtUtils::MakeMaker", "Module::Install"

•

A typical "pp"−generated "META.yml" looks like this:

•

The "par:" settings controls its runtime behavior

•

OpenPGP clear-signed manifest with SHA1 digests

Supported by "Module::Signature", "CPANPLUS" and "Module::Build"

•

A typical "SIGNATURE" looks like this:

•

Use "pp" and "cpansign" to work with signatures:

•

Framework for self-contained Web applications

Similar to Java’s "Web Application Archive" (WAR) files

•

Works with mod_perl 1.x or 2.x

•

•

Apache configuration, static files, Perl modules...

•

Supports Static, Registry and PerlRun handlers

•

Can also load all PARs under a directory

•

•

First, make a "hondah.par" from an one-liner:

•

Add this to "httpd.conf", then restart apache:

•

Test it out:

•

Instant one-liner web application that works!

•

With LWP installed, your can use remote PAR files:

•

Modules are now cached under $ENV{PAR_GLOBAL_TEMP}

•

Auto-updates with "LWP::Simple::mirror"

Download only if modified

•

Safe for offline use after the first time

•

May use "SIGNATURE" to prevent DNS-spoofing

•

•

Upgrades from a central location

•

No installers needed

•

Also known as source-hiding techniques

It is not encryption

•

Offered by PerlApp, Perl2Exe, Stunnix...

•

•

Take optree dump from memory, feed to "B::Deparse"

•

If you just want to stop a casual "grep", "deflate" already works

•

•

Bundled examples: Bleach, PodStrip and PatchContent

•

True encryption using "Crypt::*"

•

Or even _product activation_ over the internet

•

•

To get the host archive from a packed program:

•

Same thing, but with read_file():

•

Loaded PAR files are stored in %PAR::LibCache:

•

GUI toolkits often need to link with shared libraries:

•

Use "pp −−gui" on Win32 to eliminate the console window:

•

"Can’t locate Foo/Widget/Bar.pm in @INC"?

Some toolkits (notably Tk) autoloads modules without "use" or "require"

•

Hence "pp" and "Module::ScanDeps" may fail to detect them

•

Tk problems mostly fixed by now, but other toolkits may still break

•

You can work around it with "pp −M" or an explicit "require"

•

Or better, send a short test-case to "[email protected]" so we can fix it

•

Installing XS extensions from CPAN was difficult

Some platforms do not come with a compiler (Win32, MacOSX...)

•

Some headers or libraries may be missing

•

PAR.pm itself used to suffer from both problems

•

•

Users will not notice anything, except now it works

Of course, you still need to type "make par" and upload the precompiled package

•

PAR users can also install it directly with "parl −i"

•

Additional resources

Mailing list: "[email protected]"

•

Subscribe: Send a blank email to "par−[email protected]"

•

List archive: <http://nntp.x.perl.org/group/perl.par>

•

PAR::Intro: <http://search.cpan.org/dist/PAR/lib/PAR/Intro.pod>

•

Apache::PAR: <http://search.cpan.org/dist/Apache−PAR/>

•

Module::Install: <http://search.cpan.org/dist/Module−Install/>

•

•

Here begins the scary part

Grues, Dragons and Jabberwocks abound...

•

You are going to learn weird things about Perl internals

•

•

@INC code references

•

On-the-fly source filtering

•

Overriding DynaLoader::bootstrap() to handle XS modules

•

Making self-bootstrapping binary executables

•

•

DynaLoader and %INC are there since Perl 5 was born

•

PAR currently needs 5.6, but a 5.005 port is possible

•

On 1999−07−19, Ken Fox submitted a patch to P5P

To _enable using remote modules_ by putting hooks in @INC

•

It’s accepted to come in Perl 5.6, but undocumented until 5.8

•

Type "perldoc −f require" to read the nitty-gritty details

•

•

Perl 5.8 let you open a file handle to a string, so we just use that:

•

But Perl 5.6 does not have that, and I don’t want to use temp files...

•

... Undocumented features to the rescue!

It turns out that @INC hooks can return two values

•

The first is still the file handle

•

The second is a code reference for line-by-line source filtering!

•

•

But we don’t really have a filehandle for anything

•

Another undocumented feature saves the day!

•

We can actually omit the first return value altogether:

•

XS modules have dynamically loaded libraries

They cannot be loaded as part of a zip file, so we extract them out

•

Must intercept DynaLoader’s library-finding process

•

•

During the process, it calls "dl_findfile" to locate the file

•

So we install pre-hooks around both functions

•

•

If yes, extract it to a "File::Temp" temp file

The file will be automatically cleaned up when the program ends

•

It then pass the arguments to the original "bootstrap"

•

Finally, our "dl_findfile" intercepts known filenames and return it

•

The par script ($0) itself

May be in plain-text or native executable format

•

Any number of embedded files

Typically used to bootstrap PAR’s various dependencies

•

Each section begins with the magic string "FILE"

•

Length of filename in pack(’N’) format and the filename (auto/.../)

•

File length in pack(’N’) and the file’s content (not compressed)

•

•

Just a regular zip file with the magic string "PK\003\004"

•

•

A pack(’N’) number of the total length of FILE and PAR sections

•

Finally, there must be a 8−bytes magic string: "\012PAR.pm\012"

•

All we can expect is a working perl interpreter

The self-contained script *must not* use any modules at all

•

But to process PAR files, we need XS modules like Compress::Zlib

•

•

That’s what the "FILE" section in the previous slide is for

•

Load modules to memory, and write object files to disk

•

Then use a local @INC hook to load them on demand

•

•

First, try to load PerlIO::scalar and File::Temp

•

Set up an END hook to unlink all temp files up to this point

•

Load other bundled files, and look in the compressed PAR section

•

This can be much easier with a pure-perl inflate(); patches welcome!

•

Any questions, please?