Server-side PDF document generation

This article explains how to generate/modify PDFs using various methods, libraries and tools. I also introduce my PoC on generating PDFs using the PDF templates created in Word/Excel and having fixed placeholders or dynamic substitution tags (such as {FIRSTNAME} or {CURRENTDATE})

Introduction

PDF is the de facto standard for “final form documents”, it is electronic paper. Like paper, PDF combines characteristics that encompass any document, in any setting. Legal briefs, product manuals, sheet music, phone-bills, articles, construction drawings, all may be faithfully represented, integrated and exchanged with PDF. The format is created by Adobe in 1992-1993 as part of their Acrobat product. It turned over ISO in 2007. Now it is an international standard from ISO. For the static documents, everything is pretty clear: you can create a PDF manually using iText, PDFBox or Apache FOP, and upload the file to the web server. I played around with these tools, and you will find some examples in the article. In order to create a PDF based on the dynamic information, such as database contents or user input, you will possibly face to some technical difficulties rooted in the nature and limitations of PDF format. In this article, I will share them and propose the solutions. I will also present my PoC about using configurable PDF templates for creating a unique PDF by request. There are some findings on how SAP hybris uses PDF generation libraries. Read the last section for this information.

Summary

• Creating PDF from scratch
  • iText
    • Example with iText
  • PDFBox
  • Apache FOP
    • Example with FOP
• Using a template
  • Fixed placeholders
    • Example with PDFBox
  • Non-fixed placeholders
    • Using PDF templates
      • Using fields
      • Using substitution tags
        • Microsoft Office templates (DOCX, XLSX)
          • Example with Apache POI
        • HTML templates
        • PDF templates
          • Example with PDFBox
• Useful PDF Manipulation utilities
• PDF generation in SAP hybris

Creating PDF from scratch

There is a number of Java libraries and tools designed for creating PDF documents dynamically. I compared the performance of all three libraries (very roughly) and that is what I got:

iText	PDFBox	Apache FOP
1.4 sec	1.7 sec	2.8 sec

iText

iText is a mature open source library for reading and writing PDF documents. It supports fillable forms, encrypted files. I think it is a most powerful library for PDF creating and parsing, but the last version 7.0.2 is not free, it is a commercial product. 2.7.1 is GPL, that means you can use it in open source projects. 2.1.7 is MPL which means source code must be disclosed. So if you are planning to use this library, the version counts. The resulting PDF: This code uses iText 2.1.7 which license is MPL. It takes ~1.4 sec on my laptop (Dell E5470) to create a PDF shown above.

PDFBox

PDFBox is a library to create and modify PDF document on-the-fly. It offers a lot of features to generate page, read existing pdf document text and draw on blank template. See the examples on how PDFBox works below in the text. It takes ~1.7 sec on my laptop (Dell E5470) to create a PDF similar to generated by iText (see above) and Apache FOP (see below).

Apache FOP

Apache FOP (Formatting Objects Processor) is a application that converts XSL Formatting Objects (XSL-FO) files to PDF or other printable formats. Apache FOP implies that the developer creates an XML schema that describes how data objects should be organized in the final PDF. This XML schema is called XSL-FO.  Commonly this template is created by developers based on the mock-ups from designers. Any changes in the mock-up will lead to making changes in XSL-FO. Some minor changes may require a full rewrite of XSL-FO that is time consuming and non-friendly neither for developers nor business users. But there are a number of disadvantages with that approach, e.g.:

• you have to learn XSL-FO (and therefore also XPath and XSL).
• even if you know XSL-FO you do not want to be writing FO templates using a text editor.
• there is some logic that cannot be easily defined using XSL-FO. For example, grouping of data requires you to use the so called Muenchian method , which isn’t straightforward to understand.
• this approach is not good for large multi-page documents with the comprehensive structure and layout
• supporting large and sophisticated XSL-FO may be difficult, time consuming and costly
• business users will need to ask developers for any change in the template, even minor.

For example, Hybris Financial Accelerator uses Apache FOP for generating downloadable policies. This code creates the following PDF: It takes ~2.8 sec on my laptop (Dell E5470) to generate a PDF file from the XML file. Additionally, you will need some time to create an XML file from the database. This example was taken from https://github.com/bzdgn/apache-fop-example

Aspose (commercial only)

Aspose (https://www.aspose.com/products/pdf) allows you to create PDF files directly through the provided API and XML templates. It will also enable you to add PDF capabilities to your applications quickly and easily. Aspose.Words is a component which provides the capability to create/edit/manipulate/transform Microsoft Word documents. You may also generate a template MS Word document and use Mail Merge and other features to fill in data and create a final (merged) MS Word document. The library also provides the feature to save the resultant documents into PDF. This PDF can be delivered to the customer. The API looks more convenient than provided by the other libraries, but it is not for any project, primarily because of the price. However, I wasn’t able to test this stuff, because for some reason the trial request form didn’t work.

Using PDF templates

PDF can be created based on the existing template, by injecting some data into a designated areas in the document marked as placeholders. It is really useful when the template is comprehensive (as multi-page contracts and forms) and/or frequently updated. There are two options for the placeholders:

• placeholders are fixed. No link with the text they refer to.
• placeholders are not fixed. They tightly linked to the text they refer to.

For example, in the following document the fields are marked in red. Fixed placeholders lead to limitations in editing. Any changes in layout may lead to the changes in the placeholder configuration.  For example, if the following example, I edited a header of the document: However, fixed placeholders can be used for tickets: Fixed placeholders PDFBox provides the simplest way of adding custom objects to the existing PDF (“PDF template”): Once the fields have fixed positions, the changes in the template will lead to the necessity of re-positioning the fields. It is a manual work, and time-consuming for complex documents. However, it has a big advantage: you don’t need to solve a problem with the fonts and spacings (see the end of the article for the explanation of this challenge). Non-fixed placeholders For non-fixed placeholders there are two options:

• using fields (Acroform)
• using substitution tags

Using fields (Acroform)

According to Adobe, an AcroForm is “a flat PDF form that has some additional elements—the interactive fields—layered above the flat render, that allow users to enter information, and allow developers to extract data from.“. These fields can be pre-filled at the server using PDF manipulation libraries. The flaw is you will not likely able to create a PDF with the integrated Acroform fields without purchasing the package from Adobe or some other vendors.  The second problem is that your master copy of the document is likely in the Microsoft Word rather than in PDF. That means that you need to add fields (manually) every time you make changes in the document. Following is a list of some tools to create AcroForms

Using substitution tags

I tried to find ready-to-use solutions for this option, and I failed. So I decided to create my own PoC for illustrating this approach. Please be aware that this approach is experimental and there are some limitations. The idea is in using Microsoft products for creating and editing templates, such as contracts or reports. The data from the database or user input is injected into the document, specifically, in the named areas. So the approach is close to Acroform, but no additional software is required. There is a convention on the placeholders design. In the example below, the placeholder has a gray background and curly braced. The same font family and size are used for all fields. The fields have enough room for the values of any size within agreed range. The version for Excel: So these templates can be merged on the server with the dataset to create a unique PDF file. However, it is not as easy as it looks. The template can be stored on the website in the following forms:

• as PDF file
• as Microsoft Office (Word/Excel/Powerpnt) file
• as HTML file.

Microsoft Office Microsoft Word / Microsoft Excel files can be changed using Apache POI. However, in this case the output format will be DOCX, not PDF. The following code reads MyDocTemplate.docx, replaced THECLIENTNAME with “Rauf Aliev” and saves the result to GeneratedFile.docx. Unfortunately, Apache POI can’t generate PDF from DOC. The reasons are clear: converting to PDF requires a word processor engine. If you need to convert DOC to PDF, you can use Unoconv. It converts documents from and to a LibreOffice supported format. Microsoft formats and PDFs are supported.

unoconv -f pdf

MyDocument.docx

HTML to PDF You can create an HTML page and render it to PDF for printing. The simplest and very convenient solution is PhantonJS headless browser. For example, you can create a PDF of this blog using the following command:

phantomjs rasterize.js http://hybrismart.com hybrismart.pdf A4

The resulted PDF: The following example demonstrates the same with our sample contract. Originally it was created in Microsoft Word, them saved as HTML, uploaded to the server, and the server side script merged it with the data and the resulting HTML was converted to PDF using PhantomJS: For our example with the contract, you can store the template as HTML. The administrator will be able to edit it with the WYSISYG editor in backoffice, and the system will inject data into the template and create a PDF file using PhantomJS. Using PDF template The last approach is creating a PDF based on the PDF tempalte. The system uses the PDF template and it generates a final PDF by merging the template with the data. This is a screenshot from my PoC: As you see, the placeholders are replaced with the specific data. In this PoC the rules are hard-coded for demonstration purposes. The main problem with editing PDF is that once you have the PDF file, the original layout and meaning formed from text-based building blocks — including words, lines (and line breaks), paragraphs, columns, tables, headers/footers and outlines — are long gone. Once in a PDF, its content just describes how and where on the page each object should be displayed and no more else. That creates the following challenges:

• All objects in PDF are assigned to the specific location at the page. These locations are calculated by the software that was used for PDF generation (MS Word/Excel in our case)
• All words are separate objects organized into arrays.
• What is worse, the parts of words are separate objects organized into arrays.
• Even letters can be separate objects organized into arrays.
• There is no logic on how letters, words or their parts organized into arrays. It is magic.
• Objects (words, parts of words, letters) are not nicely linked with each other.
• As a rule, neighboring objects are also close/next to each other in the text.
• The characters in the objects need to be synced with the dictionary explained in the other part of PDF. If your original document doesn’t have “A” or “C”, and you decide to change “G” to “A” or “C”, you will get an empty box instead of the character.
• There is no “underlined” attribute in PDF; an underline is created by drawing a line under the text which should then appear as “underlined”.
• and so on

For example, in my demo document here are objects and arrays: For example, PDF to Text utility analyzes these objects on how close they are, and based on this information they decide if these objects need to be merged in the same text block or not. For each small text object (red boxes at the diagram above), the PDF includes co-ordinates that describe where it should be positioned on the page and how it should be displayed. In our case we need to replace the token {THE-CLIENT-NAME} with the name. In my PoC, I replace the characters of the array items to the corresponding fragments from the target string.

• “{” -> “R”
• “THE” -> “AUF”
• “-” -> ” “
• “CLIENT” -> “ALIEV “
• “-” -> ” “
• “NAME” -> ” “
• “}” -> ” “

For option #2:

• “-” -> “V” (for option #2)
• “NAME” -> “ELIE”
• “}” -> “VICH”

In PDF, these letters are stored with the individual distances (kerning), so that is why the fixed width fonts are used here. For non-fixed font, such symbols as I and W have different spacing. It is very difficult (almost impossible) to recalculate these intervals. So it is easier to go with the assumption that fixed fonts are ok for the fields. The explained approach will work only with the particular PDF generating applications and even with the particular versions. I tested it with Microsoft Office (Word, Excel, Powerpoint). It is very convinient that Office uses ASCII codes for the characters in the text objects. However, PDF standard allows using random codes if you create a proper dictionary. The explained approach won’t work with the PDFs created with non-ASCII characters. This approach was not tested on non-latin character set. Unfortunately, it makes necessary to use a hack to avoid deep change of PDF template: in order to build a proper dictionary in the PDF template and full set of characters, you need to add a line with a full character set in the same font and size as your fields to the page and make it invisible (white on white): “ABCDE….XYZ12345…/.,”. It should not be forgotten that different countries use different standards for paper, so the PDF template formatted for A4 paper size will not work nicely in the USA. If the file is generated for Letter, it can be printed in A4 too, and vice versa, but the versions will have a slightly different look because of scaling.

Useful PDF manipulation utilities

QPDF

QPDF is a PDF-to-PDF converter. It is capable of performing a variety of transformations such as linearization (also known as web optimization or fast web viewing), encryption, and decryption of PDF files. It is very useful for debugging purposes as well. You can use this utility to inject the generated page into a multi-page contract created separately using PDF merge capability of QPDF.

PDFXplorer

Very simple, but very useful utility for viewing the structure of a PDF file in a tree like view. It displays the content of stream objects as text or hex. Image streams can be displayed as image (only DCT encoded streams) or as hex. The application is free for personal and commercial use.

SAP Hybris and PDF generation

The following components in hybris are related to the PDF generation:

• Hybris Cockpit PDF Export. It is a very old module, and PDF is used for exporting data (Jasper Reports Export Service). PDF is generated using Jasper Reports PDF Exporter.
• Hybris Order PDF Export. This module is also old (Jalo). In order to export the order contents to PDF, the order is exported to XML first (CustomOrder2XML), and the XML is converted to PDF according to the schema (XML-FO). For this conversion, Apache FOP is used.
• Hybris also uses JasperReports to generate a PDF from cockpits (JRPdfExporter)
• iText 2.1.7 (MPL) is included to the hybris. However there are no signs that this library is used by hybris code.
• hybris MIME Service is capable to identify PDFs by the first bytes of the content (header). Normally, the PDF file should be started with “%PDF”. Hybris uses Medsea Mime Utils to set proper mime type when you download a media containing the PDF file.
• Financial Accelerator uses Apache FOP library for the PDF generation. Financial Accelerator is bundled with XML-FO templates for different insurance types (auto, event, life, property home owner, property renter, and travel). Good overview of classes regarding PDF document creation is on the hybris help website: https://help.hybris.com/6.3.0/hcd/8accbb0d86691014a30cb484ecd52786.html 

© Rauf Aliev, June 2017