-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Showing
5 changed files
with
707 additions
and
1,378 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -127,3 +127,4 @@ dmypy.json | |
|
|
||
| # Pyre type checker | ||
| .pyre/ | ||
| .vscode/settings.json | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,236 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 2, | ||
| "id": "c0e930be-4840-40f1-b031-34ddcc8bb933", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import spacy\n", | ||
| "from spacypdfreader import pdf_reader" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 4, | ||
| "id": "b35fe7f7-80e4-401f-8346-348acd29138a", | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "data": { | ||
| "application/vnd.jupyter.widget-view+json": { | ||
| "model_id": "4f6c96b891c1427c990d479ec5a2bcd2", | ||
| "version_major": 2, | ||
| "version_minor": 0 | ||
| }, | ||
| "text/plain": [ | ||
| "Output()" | ||
| ] | ||
| }, | ||
| "metadata": {}, | ||
| "output_type": "display_data" | ||
| } | ||
| ], | ||
| "source": [ | ||
| "nlp = spacy.load(\"en_core_web_sm\")\n", | ||
| "doc = pdf_reader(\"../tests/data/test_pdf_01.pdf\", nlp)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 6, | ||
| "id": "f72651b5-1729-42a8-977b-74f8a668f383", | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "data": { | ||
| "text/plain": [ | ||
| "Test PDF 01\n", | ||
| "\n", | ||
| "This is a simple test pdf" | ||
| ] | ||
| }, | ||
| "execution_count": 6, | ||
| "metadata": {}, | ||
| "output_type": "execute_result" | ||
| } | ||
| ], | ||
| "source": [ | ||
| "doc[0:10]" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 7, | ||
| "id": "3adc61fd-e0cb-43c4-8cb6-70fa9d51b990", | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "data": { | ||
| "text/plain": [ | ||
| "U3D or PRC and various other data formats.[15][16][17]\n", | ||
| "\n" | ||
| ] | ||
| }, | ||
| "execution_count": 7, | ||
| "metadata": {}, | ||
| "output_type": "execute_result" | ||
| } | ||
| ], | ||
| "source": [ | ||
| "doc[-10:]" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 5, | ||
| "id": "ade6938d-70b0-4aa4-9d7d-96297ec9f9bd", | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "data": { | ||
| "text/plain": [ | ||
| "Test PDF 01\n", | ||
| "\n", | ||
| "This is a simple test pdf. This is page 1.\n", | ||
| "\n", | ||
| "Portable Document Format (PDF), standardized as ISO 32000, is a file format developed by\n", | ||
| "Adobe in 1993 to present documents, including text formatting and images, in a manner\n", | ||
| "independent of application software, hardware, and operating systems.[2][3] Based on the\n", | ||
| "PostScript language, each PDF file encapsulates a complete description of a fixed-layout flat\n", | ||
| "document, including the text, fonts, vector graphics, raster images and other information needed\n", | ||
| "to display it.\n", | ||
| "\n", | ||
| "PDF was standardized as ISO 32000 in 2008.[4] The last edition as ISO 32000-2:2020 was\n", | ||
| "published in December 2020.\n", | ||
| "\n", | ||
| "PDF files may contain a variety of content besides flat text and graphics including logical\n", | ||
| "structuring elements, interactive elements such as annotations and form-fields, layers, rich\n", | ||
| "media (including video content), and three-dimensional objects using U3D or PRC, and various\n", | ||
| "other data formats. The PDF specification also provides for encryption and digital signatures, file\n", | ||
| "attachments, and metadata to enable workflows requiring these features.\n", | ||
| "\n", | ||
| "\n", | ||
| "This is page 2\n", | ||
| "\n", | ||
| "History of PDF\n", | ||
| "\n", | ||
| "Adobe Systems made the PDF specification available free of charge in 1993. In the early years\n", | ||
| "PDF was popular mainly in desktop publishing workflows, and competed with a variety of\n", | ||
| "formats such as DjVu, Envoy, Common Ground Digital Paper, Farallon Replica and even\n", | ||
| "Adobe's own PostScript format.\n", | ||
| "\n", | ||
| "PDF was a proprietary format controlled by Adobe until it was released as an open standard on\n", | ||
| "July 1, 2008, and published by the International Organization for Standardization as ISO\n", | ||
| "32000-1:2008,[5][6] at which time control of the specification passed to an ISO Committee of\n", | ||
| "volunteer industry experts. In 2008, Adobe published a Public Patent License to ISO 32000-1\n", | ||
| "granting royalty-free rights for all patents owned by Adobe that are necessary to make, use, sell,\n", | ||
| "and distribute PDF-compliant implementations.[7]\n", | ||
| "\n", | ||
| "PDF 1.7, the sixth edition of the PDF specification that became ISO 32000-1, includes some\n", | ||
| "proprietary technologies defined only by Adobe, such as Adobe XML Forms Architecture (XFA)\n", | ||
| "and JavaScript extension for Acrobat, which are referenced by ISO 32000-1 as normative and\n", | ||
| "indispensable for the full implementation of the ISO 32000-1 specification. These proprietary\n", | ||
| "technologies are not standardized and their specification is published only on Adobe's\n", | ||
| "website.[8][9][10][11][12] Many of them are also not supported by popular third-party\n", | ||
| "implementations of PDF.\n", | ||
| "\n", | ||
| "In December, 2020, the second edition of PDF 2.0, ISO 32000-2:2020, was published, including\n", | ||
| "clarifications, corrections and critical updates to normative references.[13] ISO 32000-2 does\n", | ||
| "not include any proprietary technologies as normative references.[14]\n", | ||
| "\n", | ||
| "\n", | ||
| "This is page 3\n", | ||
| "\n", | ||
| "A PDF file is often a combination of vector graphics, text, and bitmap graphics. The basic types\n", | ||
| "of content in a PDF are:\n", | ||
| "\n", | ||
| "Text stored as content streams (i.e., not encoded in plain text);\n", | ||
| "Vector graphics for illustrations and designs that consist of shapes and lines;\n", | ||
| "Raster graphics for photographs and other types of images\n", | ||
| "Multimedia objects in the document.\n", | ||
| "In later PDF revisions, a PDF document can also support links (inside document or web page),\n", | ||
| "forms, JavaScript (initially available as a plugin for Acrobat 3.0), or any other types of embedded\n", | ||
| "contents that can be handled using plug-ins.\n", | ||
| "\n", | ||
| "PDF combines three technologies:\n", | ||
| "\n", | ||
| "A subset of the PostScript page description programming language, for generating the layout\n", | ||
| "and graphics.\n", | ||
| "A font-embedding/replacement system to allow fonts to travel with the documents.\n", | ||
| "A structured storage system to bundle these elements and any associated content into a single\n", | ||
| "file, with data compression where appropriate.\n", | ||
| "PostScript is a page description language run in an interpreter to generate an image, a process\n", | ||
| "requiring many resources. It can handle graphics and standard features of programming\n", | ||
| "languages such as if statements and loop commands. PDF is largely based on PostScript but\n", | ||
| "simplified to remove flow control features like these, while graphics commands such as lineto\n", | ||
| "remain.\n", | ||
| "\n", | ||
| "Often, the PostScript-like PDF code is generated from a source PostScript file. The graphics\n", | ||
| "commands that are output by the PostScript code are collected and tokenized. Any files,\n", | ||
| "graphics, or fonts to which the document refers also are collected. Then, everything is\n", | ||
| "compressed to a single file. Therefore, the entire PostScript world (fonts, layout, measurements)\n", | ||
| "remains intact.\n", | ||
| "\n", | ||
| "As a document format, PDF has several advantages over PostScript:\n", | ||
| "\n", | ||
| "PDF contains tokenized and interpreted results of the PostScript source code, for direct\n", | ||
| "correspondence between changes to items in the PDF page description and changes to the\n", | ||
| "resulting page appearance.\n", | ||
| "PDF (from version 1.4) supports transparent graphics; PostScript does not.\n", | ||
| "PostScript is an interpreted programming language with an implicit global state, so instructions\n", | ||
| "accompanying the description of one page can affect the appearance of any following page.\n", | ||
| "Therefore, all preceding pages in a PostScript document must be processed to determine the\n", | ||
| "correct appearance of a given page, whereas each page in a PDF document is unaffected by\n", | ||
| "the others. As a result, PDF viewers allow the user to quickly jump to the final pages of a long\n", | ||
| "document, whereas a PostScript viewer needs to process all pages sequentially before being\n", | ||
| "\n", | ||
| "\n", | ||
| "able to display the destination page (unless the optional PostScript Document Structuring\n", | ||
| "Conventions have been carefully compiled and included).\n", | ||
| "PDF 1.6 supports interactive 3D documents embedded in a PDF file: 3D drawings can be\n", | ||
| "embedded using U3D or PRC and various other data formats.[15][16][17]\n", | ||
| "\n" | ||
| ] | ||
| }, | ||
| "execution_count": 5, | ||
| "metadata": {}, | ||
| "output_type": "execute_result" | ||
| } | ||
| ], | ||
| "source": [ | ||
| "doc" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "id": "69e682f0-1b2d-447c-ac59-1edad3050127", | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.9.4" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 5 | ||
| } |
Oops, something went wrong.