Scripts from geo-infra

Generate Utils

Helper class and functions to generate the configuration of a portal.

class generate_utils.Generate(portal=None, type=None, verbose=False, ssl_validation=True, config=None)[source]

Utilitary class for all generate type classes.

Parameters:
  • portal (str) – the name of the portal used to generate.
  • type (str) – the type for which to generate. Must be either dev or prod.
  • verbose (bool) – print verbose messages.
  • ssl_validation (bool) – whether or not to verify certificates for https
  • config (generate_config.GenerateConfig) – the configuration for the portal.
DEFAULT_LANGS = ('fr', 'de')

The languages to use if no languages are specified in the configuration.

clean()[source]

Remove all files and directories in the ouput dir of the portal.

copy(src, dest, copy_dir=True, copy_dir_content=False)[source]

Copy files or directories from one location to another.

Parameters:
  • src (Union[str, list, tuple, set]) – the source of the copy. If it is iterable and not a string, the function will recurse for each element. If it is a string, each element in the resulting glob is copied.
  • dest (str) – the path to destination folder.
  • copy_dir (bool) – whether or not to copy directories. If this option is true and src is a directory, src will be copied as a directory (with all its content) into dest. The result of the copy will be dest/src/. Default: True.
  • copy_dir_content (bool) – if this option is True and src is a directory, the content of src will be copied into dest. Default: False.
path(*paths, *, ext='')[source]

Alias to path()

remove(src)[source]

Remove the given path (file or directory).

The source is only removed if it is under the output directory of the current portal. It will not remove the directory of the current portal. If you try to remove something else, a warning is printed instead.

render(template_path, dest_folder, data, dest_is_file=False, extensions=None)[source]

Render a template with jinja2 and default markup.

Parameters:
  • template_path (str) – the path to the template.
  • dest_folder (str) – where to save the result.
  • data (dict) – data used to render the template.
  • dest_is_file (bool) – indicate if dest_folder is a file. Default: False.
  • extensions (list) – list of jinja2 extensions to use. Default: None.
report_errors(errors, header=None)[source]

Print a list of errors to stderr.

Parameters:
  • errors (List[str]) – list of errors to display.
  • header (str) – an optional header to print before the errors. Default: None.
save_json(filename, content)[source]

Alias to save_json()

class generate_utils.SilentCGIHTTPRequestHandler(request, client_address, server)[source]

Helper class to silence output of CGIHTTPRequestHandler.CGIHTTPRequestHandler.

Prevent requests to be logged to stdout.

generate_utils.cgi_server(address='', port=8888)[source]

Start a silent cgi server forever.

If you call this function directly, it will block the main thread until you stop it with Ctl-C.

generate_utils.is_url(path)[source]

Return true if the argument has an ‘http’ or ‘https’ scheme.

generate_utils.path(*paths, *, ext='')[source]

Join path and append extension.

Parameters:
  • paths – the list of path to join.
  • ext (str) – the extension to append at the end of the path. Default: ‘’.
generate_utils.save_json(filename, content, pretty=False)[source]

Save a python dict in a json file.

Parameters:
  • filename (str) – the path of the file in which to save the content.
  • content (dict) – the python dict object to dump as json
  • pretty (bool) – whether or not to save the json in a easily readable way. Default: False.
generate_utils.start_cgi_server(address='', port=8888)[source]

Start the cgi server in a dedicated thread.

See also

cgi_server() the function used to start the cgi server.

Generate Config

class generate_config.ConfigErrors(header, errors)

Bases: tuple

Store the configuration errors.

Parameters:
  • header (str) – a message to print before the errors.
  • errors (List[str]) – the list of found errors.
_asdict()

Return a new OrderedDict which maps field names to their values.

classmethod _make(iterable, new=<built-in method __new__ of type object>, len=<built-in function len>)

Make a new ConfigErrors object from a sequence or iterable

_replace(_self, **kwds)

Return a new ConfigErrors object replacing specified fields with new values

errors

Alias for field number 1

header

Alias for field number 0

class generate_config.ConfigFileErrors(base, file, errors)

Bases: tuple

Store errors of a config file.

Parameters:
  • base (str) – The name of the file against which the config is checked.
  • file (str) – The config file that is checked for errors.
  • errors (List[Union[ConfigSectionErrors, ConfigErrors]]) – The list of found errors.
_asdict()

Return a new OrderedDict which maps field names to their values.

classmethod _make(iterable, new=<built-in method __new__ of type object>, len=<built-in function len>)

Make a new ConfigFileErrors object from a sequence or iterable

_replace(_self, **kwds)

Return a new ConfigFileErrors object replacing specified fields with new values

base

Alias for field number 0

errors

Alias for field number 2

file

Alias for field number 1

class generate_config.ConfigSectionErrors(section, errors)

Bases: tuple

Store errors found in a section of a config file.

Parameters:
  • section (str) – the section name in which the errors are found.
  • errors (List[str]) – the list of found errors.
_asdict()

Return a new OrderedDict which maps field names to their values.

classmethod _make(iterable, new=<built-in method __new__ of type object>, len=<built-in function len>)

Make a new ConfigSectionErrors object from a sequence or iterable

_replace(_self, **kwds)

Return a new ConfigSectionErrors object replacing specified fields with new values

errors

Alias for field number 1

section

Alias for field number 0

class generate_config.GenerateConfig(type=None, portal=None, infra_dir=None, prod_git_repos_location=None)[source]

Bases: object

Read the config files and provide access to the generated config with a dict like interface.

Parameters:
  • type (str) – the type for which to generate the configuration. Default: None.
  • portal (str) – portal for which to generate the configuration. Default: None.
_check_config(dest, src, depth=0, section_check=True, section=None, errors=None)[source]

Check that the source is coherent with the destination. If incoherences are found, they are reported.

Parameters:
  • dest (dict) – the destination dict.
  • src (dict) – the source dict.
  • depth (int) – the current level of recursivity. Default: 0.
  • section_check (bool) – whether or not to check for new config sections or new keys. If depth == 0 it will report the additions as sections. If depth > 0 it will report the additions as keys. Default: True.
  • section (str) – the name of the current section. Used to provide a more hepful message in case of errors. Default: None.
  • errors (list) – the list of found errors. Default: None.
_check_portal_config_with_portal_template(portal_config, cfg_file)[source]

Verify that a portal configuration file is coherent with the template.

_check_value_type(src_key, src_value, dest, errors=None)[source]

Verify that to the source key correspond a value with the same type as the dest value for this key.

Parameters:
  • src_key (str) – the source key.
  • src_value (any) – the sourve value.
  • dest (dict) – the destination dict.
  • errors (list) – the list of found errors. Default: None.
_display_config_errors(config_errors)[source]

Display the errors on stderr’

_display_config_file_errors(config_file_errors)[source]

Display the errors of one config file on stderr.

_display_config_section_errors(config_section_errors)[source]

Display the config section errors on stderr.

_display_errors()[source]

Display all the found errors for each config file on stderr.

_format_template(location)[source]

Replace the {type}, {portal}, {domain} by their value.

_format_templates(locations)[source]

Replace {type} and {portal} by their value in each list or dict it finds.

_get_config_path(cfg_file, type='dist', prefix='')[source]

Transform a catogory of file like _common into an actual path we can open.

Parameters:
  • cfg_file (str) – category of file.
  • type (str) – type of file to get (dest, dev or prod).
_load_config()[source]

Load the configuration and store it into self._config.

It will:

  1. Load the global configuration as explained in the section about configuration

  2. Add any complementary keys to self._config that are not in the config files:

    • type
    • portal
    • prod (bool)
    • infra_dir: the absolute path to the current customer infra dir
    • infra_name: the base name of infra dir, eg customer-infra
    • mapserver_ows_host: the host of mapserver (used to generate the print configuration). Only if portal is not None.
    • prod_git_repos_location: location of the productions git repositories on the server.
_load_config_file(cfg_path, type='dist', prefix='')[source]

Load the file from the disk and parse it with the toml module.

_load_config_from_file(cfg_file, type, portal_file=False, prefix='', must_exists=True)[source]

Load the config file and override keys with those from prod or dev if needed.

Parameters:
  • cfg_file (str) – either a path to an existing file or a category of files, eg _common.
  • protal_file (bool) – if true, the config file will be checked against _template.dist.toml.
_update_config(dest, src, depth=0, section_check=True, section=None, errors=None)[source]

Recursively update a dict while checking for inconsistancies.

Parameters:
  • dest (dict) – the destination dict.
  • src (dict) – the source dict.
  • depth (int) – the current level of recursivity. Default: 0.
  • section_check (bool) – whether or not to check for new config sections or new keys. If depth == 0 it will report the additions as sections. If depth > 0 it will report the additions as keys. Default: True.
  • section (str) – the name of the current section. Used to provide a more hepful message in case of errors. Default: None.
  • errors (list) – the list of found errors. Default: None.
config

Access to the row config dict.

create_output_dirs()[source]

Create all the directories for a portal output.

These directories correspond to the dest section of the config. Keys that start with ‘_template’ and the geo_front3 subsection are ignored.

get(*args, **kwargs)[source]

Forward to self.config.get

optional_values = {'front.default_values': {'wms_list', 'wmts_list'}}

Optional keys in the configuration. If one of these keys is missing, no warning is reported.

Generate Help

class generate_help.GenerateHelpConfig(portal=None, type=None, verbose=False, ssl_validation=True, config=None)[source]

Bases: generate_utils.Generate

Generate the help website and fetch texts and images from Swisstopo.

FRONT_HELP_TEMPLATE = {'kind': 'fusiontables#sqlresponse', 'rows': [['{id}', 'Recherche de données (catalogue)', 'Sous "Changer thème" vous trouverez l\'ensemble des géodonnées disponibles classées par thèmes. Lorsque vous sélectionnez un jeu de données (en cliquant sur le cercle à droite), celui-ci est directement affiché.', '', '//help.geo.admin.ch/img/32f.jpg', 'fr', '', '', '15', 'submenu3']], 'columns': ['id', 'Title', 'Content', 'Legend', 'Image', 'lang', 'RST Content', 'HTML Content', 'sort', 'group']}

The template for help files used in the front end to display help.

HELP_GEO_ADMIN_REGEXP = re.compile('(?:https?:)?//help.geo.admin.ch/([^ ])')

Regexp used to detect the links to the Swisstopo website so we can correct them.

JEPG_TO_PNG_REGEXP = re.compile('\\.(jpg|jpeg)')

Regexp used to convert images from jpg to png.

SPECIAL_PAGES_NAME_TO_ID = {'shortcuts': 69, 'parameter': 54}

Convent the name of a special page to its id.

SPECIAL_PAGES_URLS = {'shortcuts': {'en': 'https://raw.githubusercontent.com/geoadmin/web-geoadmin-help/master/htdocs/shortcuts.html'}, 'parameter': {'fr': 'https://raw.githubusercontent.com/geoadmin/web-geoadmin-help/master/htdocs/special/parameter_fr.html', 'en': 'https://raw.githubusercontent.com/geoadmin/web-geoadmin-help/master/htdocs/special/parameter_en.html', 'de': 'https://raw.githubusercontent.com/geoadmin/web-geoadmin-help/master/htdocs/special/parameter.html'}}

Where to download the special pages.

TEXT_URLS = {'en': 'https://www.googleapis.com/fusiontables/v1/query?key=AIzaSyDT7wmEx97gAG5OnPwKyz2PnCx3yT4j7C0&sql=select+*+from+1Tx2VSM1WHZfDXzf8rweRLG1kd23AA4aw8xnZ_3c+where+col5+%3D+%27en%27+order+by+col8&_=1442242212913', 'fr': 'https://www.googleapis.com/fusiontables/v1/query?key=AIzaSyDT7wmEx97gAG5OnPwKyz2PnCx3yT4j7C0&sql=select+*+from+1Tx2VSM1WHZfDXzf8rweRLG1kd23AA4aw8xnZ_3c+where+col5+%3D+%27fr%27+order+by+col8&_=1444823149806', 'it': 'https://www.googleapis.com/fusiontables/v1/query?key=AIzaSyDT7wmEx97gAG5OnPwKyz2PnCx3yT4j7C0&sql=select+*+from+1Tx2VSM1WHZfDXzf8rweRLG1kd23AA4aw8xnZ_3c+where+col5+%3D+%27it%27+order+by+col8&_=1444823166829', 'de': 'https://www.googleapis.com/fusiontables/v1/query?key=AIzaSyDT7wmEx97gAG5OnPwKyz2PnCx3yT4j7C0&sql=select+*+from+1Tx2VSM1WHZfDXzf8rweRLG1kd23AA4aw8xnZ_3c+where+col5+%3D+%27de%27+order+by+col8&_=1444823120334'}

URLS pointing to the text in each languages.

fetch_original_content()[source]

Fetch the help content (texts and images) from Swisstopo.

  1. Fetch the special pages and format them for inclusion in the CSV files.
  2. Fetch the texts.
  3. Append the special pages.
  4. Correct the link in the texts and save them in CSV files.
  5. Fetch the images, convert them to PNG and save them.
generate()[source]

Generate the help website and its content.

  1. Copy the static files.
  2. Copy the image (swisstopo’s then the ones of the portal).
  3. Generate the help texts (for the website and for the frontend).
  4. Generate the lang files which defines which langs are available on the website.
ignores = None

Contains the id of the help texts to ignore

Generate images

class generate_images.GenerateImages(portal=None, type=None, verbose=False, ssl_validation=True, config=None)[source]

Bases: generate_utils.Generate

Copy the images for one portal.

It will fist copy the general images from src[‘img’] then the portal images from src[‘img_portal’]. All the image will be copied in dest[‘img’].

Generate JSON

Contains classes to parse WMS GetCapabilities and generate JSON config files for the frontend.

class generate_json.GenerateJsonConfig(**kwargs)[source]

Generate the layers configuration and the catalog from a GetCapabilities request.

In order to always use the lastest map file, we start a python CGI server in a thread. This server will be used for the GetCapabilities request.

check_layers_in_layers_config(layers, topic_name)[source]

Verify for each layer of a topic that is exists in the layers configuration.

Print the list of layers that don’t have a configuration on stderr.

create_catalog_json()[source]

Create the configuration of the catalog for each topic.

create_services_json()[source]

Create the list of topics.

generate()[source]

Generate the layers configuration and the catalog.

process_category(category, t)[source]

Generate the configuration for a category of the catalog.

process_layer(layer_id, t)[source]

Return the proper configuration for a layer in the catalog.

class generate_json.OwsParser(**kwargs)[source]

Parse a WMS GetCapabilities and saves the configuration of the layers in a JSON files.

LAYERS_CONFIG_KEY_TO_TRANSLATE = ('label', 'attribution', 'attributionUrl', 'legendUrl')

What information should we translate from the WMS server?

get_ows_information()[source]

Make the GetCapabilities request and process it.

process_external_layer_file(external_layer)[source]

Read the configuration of a layer from a JSON file.

process_external_layers()[source]

Process each JSON file configuring an external layer.

process_wms_layers()[source]

Process each WMS layer from from the GetCapabilities and store the revelant information for the configuration.

save_information()[source]

Save the layers configuration and the search files.

save_layers_config()[source]

Save the configuration for layers in a JSON file per language.

Save information required by sphinx to search for a layer in a tsv file.

The label of each layer is translated with generate_translations.CatalogTranslator so each user can search in his/her language.

translate(lang)[source]

Translate the value of the keys from OwsParser.LAYERS_CONFIG_KEY_TO_TRANSLATE.

Generate map files

class generate_map_files.GenerateMapFiles(portal=None, type=None, verbose=False, ssl_validation=True, config=None)[source]

Bases: generate_utils.Generate

Create the mapfiles based on the template in the portal folder and the configuration.

generate_map_files.MapInclude

alias of MapIncludes

Generate print

class generate_print.GeneratePrintConfig(portal=None, type=None, verbose=False, ssl_validation=True, config=None)[source]

Bases: generate_utils.Generate

Generate MFP configuration and copy the print logos.

Generate Translations

class generate_translations.CatalogTranslator(src_catalog_translations, delimiter=', ', quotechar='"')[source]

Bases: object

Allow the user to access function design to translate the layers catalog.

To acces to the translate function for a language, use catalogTranslatro[lang]. This function will then take the text to translate as an argument and return its translation.

Parameters:src_catalog_translations (str) – path to the CSV file that contains the translations for the catalog.
class generate_translations.Translator(files, verbose)[source]

Bases: generate_utils.Generate

Build the translation dictionnary used to translate the frontend.

get_translations(ignore_file_name=None, delimiter=', ', quotechar='"', output_folder=None, languages_to_save=None, pretty=True)[source]

Read the content of the translations files (on disk on from a URL) and save the constructed dict of translations in the specified output folder.

If a work is on the ignore list, it will not go into the constructed translation dict.

save_translations(output_folder, languages_to_save=None, pretty=True)[source]

Save the translation files in the JSON format for usage in the frontend.

Entry points

Render

Render document with jinja2 and ${var} as a template syntax.

Call this script with --help to view the full help message.

render.render(template_path, out_file_path, data, variable_start_string='${', variable_end_string='}')[source]

Render a template with specified variable start and variable end

Parameters:
  • template_path – The absolute path to the template.
  • out_file_path – The absolute path to the output file.
  • config – A generate_config.GenerateConfig instance.
  • variable_start_string – The string to look for at the start of a variable. Default: ‘${‘.
  • variable_end_string – The string to look for at the end of a variable. Default: ‘}’.

Generate

Main script used to generate the files for a portal.

Call this script with the --help argument for more details.