mirror of
https://github.com/openzim/zimit.git
synced 2025-12-31 04:23:15 +00:00
reset master branch for 2020 codebase
This commit is contained in:
renaud gaudin
parent
d178431e20
commit
15cf636ff3
17 changed files with 13 additions and 8305 deletions
3
Dockerfile
Normal file
3
Dockerfile
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
FROM debian:buster-slim
|
||||||
|
|
||||||
|
CMD ["/bin/bash"]
|
||||||
10
README.md
Normal file
10
README.md
Normal file
|
|
@ -0,0 +1,10 @@
|
||||||
|
zimit
|
||||||
|
===
|
||||||
|
|
||||||
|
Create ZIM files out of HTTP websites
|
||||||
|
|
||||||
|
# Previous version
|
||||||
|
|
||||||
|
A first version of a generic HTTP scraper was created in 2016 during the [Wikimania Esino Lario Hackathon](https://wikimania2016.wikimedia.org/wiki/Programme/Kiwix-dedicated_Hackathon).
|
||||||
|
|
||||||
|
That version is now considered outdated and [archived in `2016` branch](https://github.com/openzim/zimit/tree/2016).
|
||||||
246
README.rst
246
README.rst
|
|
@ -1,246 +0,0 @@
|
||||||
#####################################
|
|
||||||
Create ZIM files out of HTTP websites
|
|
||||||
#####################################
|
|
||||||
|
|
||||||
This project provides an API and an user interface in order to convert any
|
|
||||||
website into a Zim file.
|
|
||||||
|
|
||||||
Exposed API
|
|
||||||
###########
|
|
||||||
|
|
||||||
All APIs are talking JSON over HTTP. As such, all parameters should be sent as
|
|
||||||
stringified JSON and the Content-Type should be set to "application/json".
|
|
||||||
|
|
||||||
POST /website-zim
|
|
||||||
=================
|
|
||||||
|
|
||||||
By posting to this endpoint, you are asking the system to start a new download
|
|
||||||
of a website and a conversion into a Zim format.
|
|
||||||
|
|
||||||
Required parameters
|
|
||||||
-------------------
|
|
||||||
|
|
||||||
- **url**: URL of the website to be crawled
|
|
||||||
- **title**: Title that will be used in the created Zim file
|
|
||||||
- **email**: Email address that will get notified when the creation of the file is over
|
|
||||||
|
|
||||||
Optional parameters
|
|
||||||
-------------------
|
|
||||||
|
|
||||||
- **language**: An `ISO 639-3 <https://en.wikipedia.org/wiki/ISO_639-3>`_ code
|
|
||||||
representing the language
|
|
||||||
- **welcome**: the page that will be first shown in the Zim file
|
|
||||||
- **description**: The description that will be embedded in the Zim file
|
|
||||||
- **author**: The author of the content
|
|
||||||
|
|
||||||
Return values
|
|
||||||
-------------
|
|
||||||
|
|
||||||
- **job_id**: The job id is returned in JSON format. It can be used to know the
|
|
||||||
status of the process.
|
|
||||||
|
|
||||||
Status codes
|
|
||||||
------------
|
|
||||||
|
|
||||||
- `400 Bad Request` will be returned in case you are not respecting the
|
|
||||||
expected inputs. In case of error, have a look at the body of the response:
|
|
||||||
it contains information about what is missing.
|
|
||||||
- `201 Created` will be returned if the process started.
|
|
||||||
|
|
||||||
Exemple
|
|
||||||
-------
|
|
||||||
|
|
||||||
::
|
|
||||||
|
|
||||||
$ http POST http://0.0.0.0:6543/website-url url="https://refugeeinfo.eu/" title="Refugee Info" email="alexis@notmyidea.org"
|
|
||||||
HTTP/1.1 201 Created
|
|
||||||
|
|
||||||
{
|
|
||||||
"job": "5012abe3-bee2-4dd7-be87-39a88d76035d"
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
GET /status/{jobid}
|
|
||||||
===================
|
|
||||||
|
|
||||||
Retrieve the status of a job and displays the associated logs.
|
|
||||||
|
|
||||||
Return values
|
|
||||||
-------------
|
|
||||||
|
|
||||||
- **status**: The status of the job, it is one of 'queued', finished',
|
|
||||||
'failed', 'started' and 'deferred'.
|
|
||||||
- **log**: The logs of the job.
|
|
||||||
|
|
||||||
Status codes
|
|
||||||
------------
|
|
||||||
|
|
||||||
- `404 Not Found` will be returned in case the requested job does not exist.
|
|
||||||
- `200 OK` will be returned in any other case.
|
|
||||||
|
|
||||||
Exemple
|
|
||||||
-------
|
|
||||||
|
|
||||||
::
|
|
||||||
|
|
||||||
http GET http://0.0.0.0:6543/status/5012abe3-bee2-4dd7-be87-39a88d76035d
|
|
||||||
HTTP/1.1 200 OK
|
|
||||||
|
|
||||||
{
|
|
||||||
"log": "<snip>",
|
|
||||||
"status": "finished"
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
Okay, so how do I install it on my server?
|
|
||||||
##########################################
|
|
||||||
|
|
||||||
Currently, the best way to install it is by retrieving the sources from github
|
|
||||||
|
|
||||||
::
|
|
||||||
|
|
||||||
$ git clone https://github.com/almet/zimit.git
|
|
||||||
$ cd zimit
|
|
||||||
|
|
||||||
Create a virtual environment and install the project in it::
|
|
||||||
|
|
||||||
$ virtualenv venv
|
|
||||||
$ venv/bin/pip install -e .
|
|
||||||
|
|
||||||
Then, run it how you want, for instance with pserve::
|
|
||||||
|
|
||||||
$ venv/bin/pserve zimit.ini
|
|
||||||
|
|
||||||
|
|
||||||
In a separate process, you also need to run the worker::
|
|
||||||
|
|
||||||
$ venv/bin/rqworker
|
|
||||||
|
|
||||||
|
|
||||||
And you're ready to go. To test it::
|
|
||||||
|
|
||||||
$ http POST http://0.0.0.0:6543/website-url url="https://refugeeinfo.eu/" title="Refugee Info" email="alexis@notmyidea.org"
|
|
||||||
|
|
||||||
|
|
||||||
Debian dependencies
|
|
||||||
####################
|
|
||||||
|
|
||||||
Installing the dependencies
|
|
||||||
===========================
|
|
||||||
|
|
||||||
::
|
|
||||||
|
|
||||||
sudo apt-get install httrack libzim-dev libmagic-dev liblzma-dev libz-dev build-essential libtool libgumbo-dev redis-server automake pkg-config
|
|
||||||
|
|
||||||
Installing zimwriterfs
|
|
||||||
======================
|
|
||||||
|
|
||||||
::
|
|
||||||
|
|
||||||
git clone https://github.com/wikimedia/openzim.git
|
|
||||||
cd openzim/zimwriterfs
|
|
||||||
./autogen.sh
|
|
||||||
./configure
|
|
||||||
make
|
|
||||||
|
|
||||||
Then upgrade the path to zimwriterfs executable in zimit.ini
|
|
||||||
|
|
||||||
::
|
|
||||||
|
|
||||||
$ rqworker & pserve zimit.ini
|
|
||||||
|
|
||||||
How to deploy?
|
|
||||||
##############
|
|
||||||
|
|
||||||
There are multiple ways to deploy such service, so I'll describe how I do it
|
|
||||||
with my own best-practices.
|
|
||||||
|
|
||||||
First of all, get all the dependencies and the code. I like to have everything
|
|
||||||
available in /home/www, so let's consider this will be the case here::
|
|
||||||
|
|
||||||
$ mkdir /home/www/zimit.notmyidea.org
|
|
||||||
$ cd /home/www/zimit.notmyidea.org
|
|
||||||
$ git clone https://github.com/almet/zimit.git
|
|
||||||
|
|
||||||
Then, you can change the configuration file, by creating a new one::
|
|
||||||
|
|
||||||
$ cd zimit
|
|
||||||
$ cp zimit.ini local.ini
|
|
||||||
|
|
||||||
From there, you need to update the configuration to point to the correct
|
|
||||||
binaries and locations.
|
|
||||||
|
|
||||||
Nginx configuration
|
|
||||||
===================
|
|
||||||
|
|
||||||
::
|
|
||||||
|
|
||||||
# the upstream component nginx needs to connect to
|
|
||||||
upstream zimit_upstream {
|
|
||||||
server unix:///tmp/zimit.sock;
|
|
||||||
}
|
|
||||||
|
|
||||||
# configuration of the server
|
|
||||||
server {
|
|
||||||
listen 80;
|
|
||||||
listen [::]:80;
|
|
||||||
server_name zimit.ideascube.org;
|
|
||||||
charset utf-8;
|
|
||||||
|
|
||||||
client_max_body_size 200M;
|
|
||||||
|
|
||||||
location /zims {
|
|
||||||
alias /home/ideascube/zimit.ideascube.org/zims/;
|
|
||||||
autoindex on;
|
|
||||||
}
|
|
||||||
|
|
||||||
# Finally, send all non-media requests to the Pyramid server.
|
|
||||||
location / {
|
|
||||||
uwsgi_pass zimit_upstream;
|
|
||||||
include /var/ideascube/uwsgi_params;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
UWSGI configuration
|
|
||||||
===================
|
|
||||||
|
|
||||||
::
|
|
||||||
|
|
||||||
[uwsgi]
|
|
||||||
uid = ideascube
|
|
||||||
gid = ideascube
|
|
||||||
chdir = /home/ideascube/zimit.ideascube.org/zimit/
|
|
||||||
ini = /home/ideascube/zimit.ideascube.org/zimit/local.ini
|
|
||||||
# the virtualenv (full path)
|
|
||||||
home = /home/ideascube/zimit.ideascube.org/venv/
|
|
||||||
|
|
||||||
# process-related settings
|
|
||||||
# master
|
|
||||||
master = true
|
|
||||||
# maximum number of worker processes
|
|
||||||
processes = 4
|
|
||||||
# the socket (use the full path to be safe
|
|
||||||
socket = /tmp/zimit.sock
|
|
||||||
# ... with appropriate permissions - may be needed
|
|
||||||
chmod-socket = 666
|
|
||||||
# stats = /tmp/ideascube.stats.sock
|
|
||||||
# clear environment on exit
|
|
||||||
vacuum = true
|
|
||||||
plugins = python
|
|
||||||
|
|
||||||
|
|
||||||
supervisord configuration
|
|
||||||
=========================
|
|
||||||
|
|
||||||
::
|
|
||||||
|
|
||||||
[program:zimit-worker]
|
|
||||||
command=/home/ideascube/zimit.ideascube.org/venv/bin/rqworker
|
|
||||||
directory=/home/ideascube/zimit.ideascube.org/zimit/
|
|
||||||
user=www-data
|
|
||||||
autostart=true
|
|
||||||
autorestart=true
|
|
||||||
redirect_stderr=true
|
|
||||||
|
|
||||||
That's it!
|
|
||||||
24
app.wsgi
24
app.wsgi
|
|
@ -1,24 +0,0 @@
|
||||||
try:
|
|
||||||
import ConfigParser as configparser
|
|
||||||
except ImportError:
|
|
||||||
import configparser
|
|
||||||
import logging.config
|
|
||||||
import os
|
|
||||||
|
|
||||||
from zimit import main
|
|
||||||
|
|
||||||
here = os.path.dirname(__file__)
|
|
||||||
|
|
||||||
ini_path = os.environ.get('ZIMIT_INI')
|
|
||||||
if ini_path is None:
|
|
||||||
ini_path = os.path.join(here, 'local.ini')
|
|
||||||
|
|
||||||
# Set up logging
|
|
||||||
logging.config.fileConfig(ini_path)
|
|
||||||
|
|
||||||
# Parse config and create WSGI app
|
|
||||||
config = configparser.ConfigParser()
|
|
||||||
config.read(ini_path)
|
|
||||||
|
|
||||||
application = main(config.items('DEFAULT'), **dict(config.items('app:main'
|
|
||||||
)))
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
.alertify-logs>*{padding:12px 24px;color:#fff;box-shadow:0 2px 5px 0 rgba(0,0,0,.2);border-radius:1px}.alertify-logs>*,.alertify-logs>.default{background:rgba(0,0,0,.8)}.alertify-logs>.error{background:rgba(244,67,54,.8)}.alertify-logs>.success{background:rgba(76,175,80,.9)}.alertify{position:fixed;background-color:rgba(0,0,0,.3);left:0;right:0;top:0;bottom:0;width:100%;height:100%;z-index:1}.alertify.hide{opacity:0;pointer-events:none}.alertify,.alertify.show{box-sizing:border-box;transition:all .33s cubic-bezier(.25,.8,.25,1)}.alertify,.alertify *{box-sizing:border-box}.alertify .dialog{padding:12px}.alertify .alert,.alertify .dialog{width:100%;margin:0 auto;position:relative;top:50%;transform:translateY(-50%)}.alertify .alert>*,.alertify .dialog>*{width:400px;max-width:95%;margin:0 auto;text-align:center;padding:12px;background:#fff;box-shadow:0 2px 4px -1px rgba(0,0,0,.14),0 4px 5px 0 rgba(0,0,0,.098),0 1px 10px 0 rgba(0,0,0,.084)}.alertify .alert .msg,.alertify .dialog .msg{padding:12px;margin-bottom:12px;margin:0;text-align:left}.alertify .alert input:not(.form-control),.alertify .dialog input:not(.form-control){margin-bottom:15px;width:100%;font-size:100%;padding:12px}.alertify .alert input:not(.form-control):focus,.alertify .dialog input:not(.form-control):focus{outline-offset:-2px}.alertify .alert nav,.alertify .dialog nav{text-align:right}.alertify .alert nav button:not(.btn):not(.pure-button):not(.md-button):not(.mdl-button),.alertify .dialog nav button:not(.btn):not(.pure-button):not(.md-button):not(.mdl-button){background:transparent;box-sizing:border-box;color:rgba(0,0,0,.87);position:relative;outline:0;border:0;display:inline-block;-ms-flex-align:center;-ms-grid-row-align:center;align-items:center;padding:0 6px;margin:6px 8px;line-height:36px;min-height:36px;white-space:nowrap;min-width:88px;text-align:center;text-transform:uppercase;font-size:14px;text-decoration:none;cursor:pointer;border:1px solid transparent;border-radius:2px}.alertify .alert nav button:not(.btn):not(.pure-button):not(.md-button):not(.mdl-button):active,.alertify .alert nav button:not(.btn):not(.pure-button):not(.md-button):not(.mdl-button):hover,.alertify .dialog nav button:not(.btn):not(.pure-button):not(.md-button):not(.mdl-button):active,.alertify .dialog nav button:not(.btn):not(.pure-button):not(.md-button):not(.mdl-button):hover{background-color:rgba(0,0,0,.05)}.alertify .alert nav button:not(.btn):not(.pure-button):not(.md-button):not(.mdl-button):focus,.alertify .dialog nav button:not(.btn):not(.pure-button):not(.md-button):not(.mdl-button):focus{border:1px solid rgba(0,0,0,.1)}.alertify .alert nav button.btn,.alertify .dialog nav button.btn{margin:6px 4px}.alertify-logs{position:fixed;z-index:1}.alertify-logs.bottom,.alertify-logs:not(.top){bottom:16px}.alertify-logs.left,.alertify-logs:not(.right){left:16px}.alertify-logs.left>*,.alertify-logs:not(.right)>*{float:left;transform:translateZ(0);height:auto}.alertify-logs.left>.show,.alertify-logs:not(.right)>.show{left:0}.alertify-logs.left>*,.alertify-logs.left>.hide,.alertify-logs:not(.right)>*,.alertify-logs:not(.right)>.hide{left:-110%}.alertify-logs.right{right:16px}.alertify-logs.right>*{float:right;transform:translateZ(0)}.alertify-logs.right>.show{right:0;opacity:1}.alertify-logs.right>*,.alertify-logs.right>.hide{right:-110%;opacity:0}.alertify-logs.top{top:0}.alertify-logs>*{box-sizing:border-box;transition:all .4s cubic-bezier(.25,.8,.25,1);position:relative;clear:both;backface-visibility:hidden;perspective:1000;max-height:0;margin:0;padding:0;overflow:hidden;opacity:0;pointer-events:none}.alertify-logs>.show{margin-top:12px;opacity:1;max-height:1000px;padding:12px;pointer-events:auto}
|
|
||||||
File diff suppressed because one or more lines are too long
7523
app/assets/bootstrap.css
vendored
7523
app/assets/bootstrap.css
vendored
File diff suppressed because it is too large
Load diff
|
|
@ -1,84 +0,0 @@
|
||||||
<!DOCTYPE html>
|
|
||||||
|
|
||||||
<head>
|
|
||||||
</head>
|
|
||||||
<link rel="stylesheet" href="./assets/bootstrap.css">
|
|
||||||
<link rel="stylesheet" href="./assets/alertify.css">
|
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
||||||
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
|
||||||
<meta http-equiv="content-type" content="text/html; charset=utf-8">
|
|
||||||
<title>Zimit — Create a zim archive out of a website URL</title>
|
|
||||||
|
|
||||||
<meta charset="utf-8" />
|
|
||||||
<body>
|
|
||||||
<div class="navbar navbar-default navbar-static-top">
|
|
||||||
<div class="container">
|
|
||||||
<div class="navbar-header">
|
|
||||||
<a class="navbar-brand" href="#">Zim it!</a>
|
|
||||||
</div>
|
|
||||||
<div class="navbar-collapse collapse">
|
|
||||||
<ul class="nav navbar-nav navbar-right">
|
|
||||||
<li><a href="http://www.openzim.org/wiki/Mission">Our values</a></li>
|
|
||||||
</ul>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
<div class="container">
|
|
||||||
<form action="#" id="zimcreator" onSubmit="submitForm()">
|
|
||||||
<div class="form-group field field-object">
|
|
||||||
<fieldset>
|
|
||||||
<div class="form-group field field-string">
|
|
||||||
<label class="control-label" for="url">Website URL</label>
|
|
||||||
<input id="url" label="Website URL" placeholder="https://google.com" class="form-control" type="url">
|
|
||||||
</div>
|
|
||||||
<div class="form-group field field-string">
|
|
||||||
<label class="control-label" for="url">Zim Title</label>
|
|
||||||
<input id="title" label="Website URL" placeholder="A great website" class="form-control" type="text">
|
|
||||||
</div>
|
|
||||||
<div class="form-group field field-string">
|
|
||||||
<label class="control-label" for="url">Enter an email to be notified when this is finished</label>
|
|
||||||
<input id="email" label="Email" placeholder="john@doe.com" class="form-control" type="email">
|
|
||||||
</div>
|
|
||||||
</fieldset>
|
|
||||||
</div>
|
|
||||||
<p>
|
|
||||||
<button type="submit" class="btn btn-info">Create the Zim file!</button>
|
|
||||||
</p>
|
|
||||||
</form>
|
|
||||||
<p>
|
|
||||||
This is a <a href="http://www.openzim.org/wiki/OpenZIM">Zim</a> creator. Enter the <em>url</em> of the website you want ton turn in a zim file, a <em>title</em> and click on <em>Create zim File</em>
|
|
||||||
</p>
|
|
||||||
<p>Enjoy !</p>
|
|
||||||
</div>
|
|
||||||
<script src="./assets/alertify.js"></script>
|
|
||||||
<script type="text/javascript">
|
|
||||||
|
|
||||||
function getField(field) {
|
|
||||||
return document.forms['zimcreator'].elements[field].value;
|
|
||||||
}
|
|
||||||
|
|
||||||
function submitForm() {
|
|
||||||
var content = {
|
|
||||||
url: getField('url'),
|
|
||||||
title: getField('title'),
|
|
||||||
email: getField('email'),
|
|
||||||
}
|
|
||||||
fetch("/website-zim", {
|
|
||||||
method: "POST",
|
|
||||||
body: JSON.stringify(content),
|
|
||||||
headers: {'Content-Type': 'application/json'}
|
|
||||||
}).then(function (result) {
|
|
||||||
if (result.status >= 400) {
|
|
||||||
alertify.error("The server wasn't able to start the job, please check your inputs.");
|
|
||||||
} else {
|
|
||||||
alertify.success("The job has been submitted! You'll receive an email when it's finished.");
|
|
||||||
}
|
|
||||||
})
|
|
||||||
.catch(function (error) {
|
|
||||||
alertify.error("Sorry, we weren't able to join the server. This is usually due to connectivity issues.");
|
|
||||||
});
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
</script>
|
|
||||||
|
|
||||||
</body>
|
|
||||||
BIN
favicon.ico
BIN
favicon.ico
Binary file not shown.
|
Before Width: | Height: | Size: 9.1 KiB |
33
setup.py
33
setup.py
|
|
@ -1,33 +0,0 @@
|
||||||
import os
|
|
||||||
from setuptools import setup, find_packages
|
|
||||||
|
|
||||||
here = os.path.abspath(os.path.dirname(__file__))
|
|
||||||
|
|
||||||
with open(os.path.join(here, 'README.rst')) as f:
|
|
||||||
README = f.read()
|
|
||||||
|
|
||||||
|
|
||||||
setup(name='zimit',
|
|
||||||
version=0.1,
|
|
||||||
description='zimit',
|
|
||||||
long_description=README,
|
|
||||||
classifiers=[
|
|
||||||
"Programming Language :: Python",
|
|
||||||
"Framework :: Pylons",
|
|
||||||
"Topic :: Internet :: WWW/HTTP",
|
|
||||||
"Topic :: Internet :: WWW/HTTP :: WSGI :: Application"
|
|
||||||
],
|
|
||||||
keywords="web services",
|
|
||||||
author='',
|
|
||||||
author_email='',
|
|
||||||
url='',
|
|
||||||
packages=find_packages(),
|
|
||||||
include_package_data=True,
|
|
||||||
zip_safe=False,
|
|
||||||
install_requires=['cornice', 'waitress', 'rq', 'colander',
|
|
||||||
'python-slugify', 'pyramid_mailer'],
|
|
||||||
entry_points="""\
|
|
||||||
[paste.app_factory]
|
|
||||||
main=zimit:main
|
|
||||||
""",
|
|
||||||
paster_plugins=['pyramid'])
|
|
||||||
62
zimit.ini
62
zimit.ini
|
|
@ -1,62 +0,0 @@
|
||||||
[app:main]
|
|
||||||
use = egg:zimit
|
|
||||||
|
|
||||||
zimit.zimwriterfs_bin = /home/alexis/dev/openzim/zimwriterfs/zimwriterfs
|
|
||||||
zimit.httrack_bin = /usr/bin/httrack
|
|
||||||
zimit.output_location = /home/alexis/dev/zimit/zims
|
|
||||||
zimit.output_url = http://zimit.notmyidea.org/zims
|
|
||||||
|
|
||||||
mail.host = localhost
|
|
||||||
mail.port = 2525
|
|
||||||
mail.default_sender = zimit@notmyidea.org
|
|
||||||
|
|
||||||
pyramid.includes =
|
|
||||||
pyramid_mailer
|
|
||||||
|
|
||||||
[server:main]
|
|
||||||
use = egg:waitress#main
|
|
||||||
host = 0.0.0.0
|
|
||||||
port = 6543
|
|
||||||
|
|
||||||
# Begin logging configuration
|
|
||||||
|
|
||||||
[uwsgi]
|
|
||||||
wsgi-file = app.wsgi
|
|
||||||
http-socket = :8000
|
|
||||||
enable-threads = true
|
|
||||||
master = true
|
|
||||||
processes = 1
|
|
||||||
virtualenv = .
|
|
||||||
module = zimit
|
|
||||||
lazy = true
|
|
||||||
lazy-apps = true
|
|
||||||
|
|
||||||
|
|
||||||
[loggers]
|
|
||||||
keys = root, gplayproxy
|
|
||||||
|
|
||||||
[handlers]
|
|
||||||
keys = console
|
|
||||||
|
|
||||||
[formatters]
|
|
||||||
keys = generic
|
|
||||||
|
|
||||||
[logger_root]
|
|
||||||
level = INFO
|
|
||||||
handlers = console
|
|
||||||
|
|
||||||
[logger_gplayproxy]
|
|
||||||
level = DEBUG
|
|
||||||
handlers =
|
|
||||||
qualname = gplayproxy
|
|
||||||
|
|
||||||
[handler_console]
|
|
||||||
class = StreamHandler
|
|
||||||
args = (sys.stderr,)
|
|
||||||
level = NOTSET
|
|
||||||
formatter = generic
|
|
||||||
|
|
||||||
[formatter_generic]
|
|
||||||
format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s
|
|
||||||
|
|
||||||
# End logging configuration
|
|
||||||
|
|
@ -1,25 +0,0 @@
|
||||||
from pyramid.config import Configurator
|
|
||||||
from pyramid.events import NewRequest
|
|
||||||
from pyramid.static import static_view
|
|
||||||
|
|
||||||
from redis import Redis
|
|
||||||
from rq import Queue
|
|
||||||
|
|
||||||
|
|
||||||
def main(global_config, **settings):
|
|
||||||
config = Configurator(settings=settings)
|
|
||||||
config.registry.queue = Queue(connection=Redis())
|
|
||||||
|
|
||||||
def attach_objects_to_request(event):
|
|
||||||
event.request.queue = config.registry.queue
|
|
||||||
|
|
||||||
config.add_subscriber(attach_objects_to_request, NewRequest)
|
|
||||||
|
|
||||||
config.include("cornice")
|
|
||||||
config.include('pyramid_mailer')
|
|
||||||
config.scan("zimit.views")
|
|
||||||
|
|
||||||
static = static_view('../app', use_subpath=True, index='index.html')
|
|
||||||
config.add_route('catchall_static', '/app/*subpath')
|
|
||||||
config.add_view(static, route_name="catchall_static")
|
|
||||||
return config.make_wsgi_app()
|
|
||||||
146
zimit/creator.py
146
zimit/creator.py
|
|
@ -1,146 +0,0 @@
|
||||||
import os
|
|
||||||
import os.path
|
|
||||||
import shutil
|
|
||||||
import tempfile
|
|
||||||
import urlparse
|
|
||||||
|
|
||||||
from slugify import slugify
|
|
||||||
|
|
||||||
from zimit import utils
|
|
||||||
|
|
||||||
HTTRACK_BIN = "/usr/bin/httrack"
|
|
||||||
DEFAULT_AUTHOR = "ZimIt"
|
|
||||||
|
|
||||||
|
|
||||||
class ZimCreator(object):
|
|
||||||
"""A synchronous zim creator, using HTTrack to spider websites and
|
|
||||||
zimwriterfs to create the zim files.
|
|
||||||
|
|
||||||
Please note that every operation is blocking the interpretor. As such, it
|
|
||||||
is recommended to run this operation in a worker if invoked from a website
|
|
||||||
view / controller.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, zimwriterfs_bin, output_location,
|
|
||||||
author=DEFAULT_AUTHOR, httrack_bin=HTTRACK_BIN,
|
|
||||||
log_file=None, max_download_speed=25000):
|
|
||||||
self.output_location = output_location
|
|
||||||
self.author = author
|
|
||||||
self.zimwriterfs_bin = zimwriterfs_bin
|
|
||||||
self.httrack_bin = httrack_bin
|
|
||||||
self.log_file = log_file
|
|
||||||
self.max_download_speed = max_download_speed
|
|
||||||
|
|
||||||
utils.ensure_paths_exists(
|
|
||||||
self.zimwriterfs_bin,
|
|
||||||
self.httrack_bin,
|
|
||||||
self.output_location)
|
|
||||||
|
|
||||||
def _spawn(self, cmd):
|
|
||||||
return utils.spawn(cmd, self.log_file)
|
|
||||||
|
|
||||||
def download_website(self, url, destination_path):
|
|
||||||
"""Downloads the website using HTTrack and wait for the results to
|
|
||||||
be available before returning.
|
|
||||||
|
|
||||||
:param url:
|
|
||||||
The entry URL of the website to retrieve.
|
|
||||||
|
|
||||||
:param destination_path:
|
|
||||||
The absolute location of a folder where the files will be written.
|
|
||||||
"""
|
|
||||||
options = {
|
|
||||||
"path": destination_path,
|
|
||||||
"max-rate": self.max_download_speed,
|
|
||||||
"keep-alive": None,
|
|
||||||
"robots": 0,
|
|
||||||
"near": None,
|
|
||||||
}
|
|
||||||
|
|
||||||
self._spawn(utils.get_command(self.httrack_bin, url, **options))
|
|
||||||
|
|
||||||
def prepare_website_folder(self, url, input_location):
|
|
||||||
"""Prepare the website files to make them ready to be embedded in a zim
|
|
||||||
file.
|
|
||||||
|
|
||||||
:returns:
|
|
||||||
the absolute location of the website folder, ready to be embedded.
|
|
||||||
"""
|
|
||||||
netloc = urlparse.urlparse(url).netloc.replace(":", "_")
|
|
||||||
website_folder = os.path.join(input_location, netloc)
|
|
||||||
if not os.path.isdir(website_folder):
|
|
||||||
message = "Unable to find the website folder! %s" % website_folder
|
|
||||||
raise Exception(message)
|
|
||||||
shutil.copy('./favicon.ico', website_folder)
|
|
||||||
return website_folder
|
|
||||||
|
|
||||||
def create_zim(self, input_location, output_name, zim_options):
|
|
||||||
"""Create a zim file out of an existing folder on disk.
|
|
||||||
|
|
||||||
:param input_location:
|
|
||||||
The absolute location of the files to be bundled in the zim file.
|
|
||||||
:param output_name:
|
|
||||||
The name to use to create the zim file.
|
|
||||||
:param options:
|
|
||||||
Options to pass to the zim creator.
|
|
||||||
"""
|
|
||||||
|
|
||||||
zim_options.update({
|
|
||||||
'bin': self.zimwriterfs_bin,
|
|
||||||
'location': input_location,
|
|
||||||
'output': os.path.join(self.output_location, output_name),
|
|
||||||
'icon': 'favicon.ico',
|
|
||||||
'publisher': self.author,
|
|
||||||
})
|
|
||||||
|
|
||||||
# Spawn zimwriterfs with the correct options.
|
|
||||||
options = (
|
|
||||||
'{bin} -w "{welcome}" -l "{language}" -t "{title}"'
|
|
||||||
' -d "{description}" -f {icon} -c "{author}"'
|
|
||||||
' -p "{publisher}" {location} {output}'
|
|
||||||
).format(**zim_options)
|
|
||||||
self._spawn(options)
|
|
||||||
return output_name
|
|
||||||
|
|
||||||
def create_zim_from_website(self, url, zim_options):
|
|
||||||
"""Create a zim file from a website. It might take some time.
|
|
||||||
|
|
||||||
The name of the generated zim file is a slugified version of its URL.
|
|
||||||
|
|
||||||
:param url:
|
|
||||||
the URL of the website to download.
|
|
||||||
|
|
||||||
:param zim_options:
|
|
||||||
A dictionary of options to use when generating the Zim file. They
|
|
||||||
are title, language, welcome and description.
|
|
||||||
|
|
||||||
:returns:
|
|
||||||
the name of the generated zim_file (relative to the output_folder)
|
|
||||||
"""
|
|
||||||
temporary_location = tempfile.mkdtemp("zimit")
|
|
||||||
self.download_website(url, temporary_location)
|
|
||||||
website_folder = self.prepare_website_folder(url, temporary_location)
|
|
||||||
output_name = "{slug}.zim".format(slug=slugify(url))
|
|
||||||
zim_file = self.create_zim(website_folder, output_name, zim_options)
|
|
||||||
return zim_file
|
|
||||||
|
|
||||||
|
|
||||||
def load_from_settings(settings, log_file=None):
|
|
||||||
"""Load the ZimCreator object from the given pyramid settings, converting
|
|
||||||
them to actual parameters.
|
|
||||||
|
|
||||||
This is a convenience function for people wanting to create a ZimCreator
|
|
||||||
out of a ini file compatible with the pyramid framework.
|
|
||||||
|
|
||||||
:param settings: the dictionary of settings.
|
|
||||||
"""
|
|
||||||
if 'zimit.zimwriterfs_bin' not in settings:
|
|
||||||
raise ValueError('Please define zimit.zimwriterfs_bin config.')
|
|
||||||
|
|
||||||
return ZimCreator(
|
|
||||||
zimwriterfs_bin=settings['zimit.zimwriterfs_bin'],
|
|
||||||
httrack_bin=settings.get('zimit.httrack_bin'),
|
|
||||||
output_location=settings.get('zimit.output_location'),
|
|
||||||
author=settings.get('zimit.default_author'),
|
|
||||||
log_file=log_file
|
|
||||||
)
|
|
||||||
|
|
@ -1,42 +0,0 @@
|
||||||
from pyramid_mailer.message import Attachment, Message
|
|
||||||
from pyramid_mailer import Mailer
|
|
||||||
|
|
||||||
|
|
||||||
def send_zim_url(settings, email, zim_url):
|
|
||||||
"""Send an email with a link to one zim file.
|
|
||||||
|
|
||||||
:param settings:
|
|
||||||
A pyramid settings object, used by pyramid_mailer.
|
|
||||||
:param email:
|
|
||||||
The email of the recipient.
|
|
||||||
:param zim_url:
|
|
||||||
The URL of the zim file.
|
|
||||||
"""
|
|
||||||
mailer = Mailer.from_settings(settings)
|
|
||||||
msg = ZimReadyMessage(email, zim_url)
|
|
||||||
mailer.send_immediately(msg)
|
|
||||||
|
|
||||||
|
|
||||||
class ZimReadyMessage(Message):
|
|
||||||
def __init__(self, email, zim_link):
|
|
||||||
subject = "[ZimIt!] Your zimfile is ready!"
|
|
||||||
|
|
||||||
bdata = """
|
|
||||||
Hi,
|
|
||||||
|
|
||||||
You have asked for the creation of a zim file, and it is now ready !
|
|
||||||
|
|
||||||
You can access it at the following URL:
|
|
||||||
|
|
||||||
{zim_link}
|
|
||||||
|
|
||||||
Cheers,
|
|
||||||
ZimIt.
|
|
||||||
""".format(zim_link=zim_link)
|
|
||||||
hdata = bdata
|
|
||||||
|
|
||||||
body = Attachment(data=bdata, transfer_encoding="quoted-printable")
|
|
||||||
html = Attachment(data=hdata, transfer_encoding="quoted-printable")
|
|
||||||
|
|
||||||
super(ZimReadyMessage, self).__init__(
|
|
||||||
subject=subject, body=body, html=html, recipients=[email])
|
|
||||||
|
|
@ -1,35 +0,0 @@
|
||||||
import os
|
|
||||||
import shlex
|
|
||||||
import subprocess
|
|
||||||
|
|
||||||
|
|
||||||
def spawn(cmd, logfile=None):
|
|
||||||
"""Quick shortcut to spawn a command on the filesystem"""
|
|
||||||
if logfile is not None:
|
|
||||||
with open(logfile, "a+") as f:
|
|
||||||
prepared_cmd = shlex.split("stdbuf -o0 %s" % cmd)
|
|
||||||
process = subprocess.Popen(prepared_cmd, stdout=f)
|
|
||||||
else:
|
|
||||||
prepared_cmd = shlex.split(cmd)
|
|
||||||
process = subprocess.Popen(prepared_cmd)
|
|
||||||
process.wait()
|
|
||||||
return process
|
|
||||||
|
|
||||||
|
|
||||||
def ensure_paths_exists(*paths):
|
|
||||||
for path in paths:
|
|
||||||
if not os.path.exists(path):
|
|
||||||
msg = '%s does not exist.' % path
|
|
||||||
raise OSError(msg)
|
|
||||||
|
|
||||||
|
|
||||||
def get_command(cmd, *params, **options):
|
|
||||||
prepared_options = []
|
|
||||||
for key, value in options.items():
|
|
||||||
if value is None:
|
|
||||||
opt = "--%s" % key
|
|
||||||
else:
|
|
||||||
opt = "--%s=%s" % (key, value)
|
|
||||||
prepared_options.append(opt)
|
|
||||||
|
|
||||||
return " ".join((cmd, " ".join(params), " ".join(prepared_options)))
|
|
||||||
|
|
@ -1,63 +0,0 @@
|
||||||
import os
|
|
||||||
|
|
||||||
from cornice import Service
|
|
||||||
from colander import MappingSchema, SchemaNode, String
|
|
||||||
from pyramid.httpexceptions import HTTPTemporaryRedirect, HTTPNotFound
|
|
||||||
|
|
||||||
from zimit.worker import create_zim
|
|
||||||
|
|
||||||
website = Service(name='website', path='/website-zim')
|
|
||||||
home = Service(name='home', path='/')
|
|
||||||
status = Service(name='status', path='/status/{id}')
|
|
||||||
|
|
||||||
|
|
||||||
@home.get()
|
|
||||||
def redirect_to_app(request):
|
|
||||||
raise HTTPTemporaryRedirect("/app/index.html")
|
|
||||||
|
|
||||||
|
|
||||||
class WebSiteSchema(MappingSchema):
|
|
||||||
url = SchemaNode(String(), location="body", type='str')
|
|
||||||
title = SchemaNode(String(), location="body", type='str')
|
|
||||||
email = SchemaNode(String(), location="body", type='str')
|
|
||||||
description = SchemaNode(String(), default="-",
|
|
||||||
location="body", type='str')
|
|
||||||
author = SchemaNode(String(), default=None,
|
|
||||||
location="body", type='str')
|
|
||||||
welcome = SchemaNode(String(), default="index.html",
|
|
||||||
location="body", type='str')
|
|
||||||
language = SchemaNode(String(), default="eng",
|
|
||||||
location="body", type='str')
|
|
||||||
|
|
||||||
|
|
||||||
@website.post(schema=WebSiteSchema)
|
|
||||||
def crawl_new_website(request):
|
|
||||||
job = request.queue.enqueue(
|
|
||||||
create_zim,
|
|
||||||
request.registry.settings,
|
|
||||||
request.validated,
|
|
||||||
timeout=1800)
|
|
||||||
request.response.status_code = 201
|
|
||||||
return {
|
|
||||||
'job_id': job.id
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@status.get()
|
|
||||||
def display_status(request):
|
|
||||||
job = request.queue.fetch_job(request.matchdict["id"])
|
|
||||||
if job is None:
|
|
||||||
raise HTTPNotFound()
|
|
||||||
|
|
||||||
log_dir = request.registry.settings.get('zimit.logdir', '/tmp')
|
|
||||||
log_file = os.path.join(log_dir, "%s.log" % job.id)
|
|
||||||
|
|
||||||
log_content = None
|
|
||||||
if os.path.exists(log_file):
|
|
||||||
with open(log_file) as f:
|
|
||||||
log_content = f.read()
|
|
||||||
|
|
||||||
return {
|
|
||||||
"status": job.status,
|
|
||||||
"log": log_content
|
|
||||||
}
|
|
||||||
|
|
@ -1,20 +0,0 @@
|
||||||
import os
|
|
||||||
import urlparse
|
|
||||||
|
|
||||||
from rq import get_current_job
|
|
||||||
|
|
||||||
from zimit.mailer import send_zim_url
|
|
||||||
from zimit.creator import load_from_settings
|
|
||||||
|
|
||||||
|
|
||||||
def create_zim(settings, options):
|
|
||||||
"""Call the zim creator and the mailer when it is finished.
|
|
||||||
"""
|
|
||||||
job = get_current_job()
|
|
||||||
log_dir = settings.get('zimit.logdir', '/tmp')
|
|
||||||
log_file = os.path.join(log_dir, "%s.log" % job.id)
|
|
||||||
zim_creator = load_from_settings(settings, log_file)
|
|
||||||
zim_file = zim_creator.create_zim_from_website(options['url'], options)
|
|
||||||
output_url = settings.get('zimit.output_url')
|
|
||||||
zim_url = urlparse.urljoin(output_url, zim_file)
|
|
||||||
send_zim_url(settings, options['email'], zim_url)
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue