Operations on source trees
Author(s): Jose F. Morales, The Ciao group.This module defines predicates to operate on directory trees containing source code (source trees).
The possible operations include enumerating, copying, and cleaning files or selection of files based on filters. See source_filter/1 for a description of the available filters.
Details of the walk algorithm
The walk algorithm (see internal walk/3) traverses a filtered directory structure and invokes actions (see internal action_hook/3, walk_action/1) parameterized by walk events (see internal walk_event/1) before entering a directory (enter event), when a file is processed (file event), and once the files in a directory are processed (exit event).
Filters restrict the walk to selected parts of the file tree. Filters are decomposed into formulas of basic filters, where a basic filter is a property of a regular file or directory (e.g., based on its name or path).
A normalized filter (see internal norm_filter/2) is given by the tuple of formulas (WalkP, DirP, FileP), where each component represents the following conditions for the walk algorithm:
- WalkP
- walk into the directory
- DirP
- treat directory files
- FileP
- treat the file (a regular file or a directory not marked for walk)
Given a base directory, the walk algorithm will perform the following operation on each file F:
if F is a directory and WalkP(F): if DirP(F): Action 'enter' on F call recursively for each file in F Action 'exit' on F else: if FileP(F): Action 'file' on F
Efficiency and memory usage
- Evaluation of filter checks is optimized to minimize accesses to the filesystem.
- This code depends on the regular expression library, which is currently not optimized for performance.
- Since file names are encoded as atoms, this code may suffer from exhaustion of the atom table on large file trees.
Examples
Find cleanable files in the current directory (recursively):
?- current_file_find([cleanable(src)], '.', X).
Find distributable packages:
?- current_file_find([proj(distributable), srctype(package)], '.', X).
Usage and interface
- Library usage:
:- use_module(library(source_tree)). - Exports:
- Imports:
- System library modules:
system, pathnames, lists, sort, aggregates, regexp_code, system_extra, paths_extra, config_common, file_utils, read. - Packages:
prelude, nonpure, condcomp, assertions, fsyntax, hiord, regtypes, isomodes.
- System library modules:
Documentation on exports
Usage:current_file_find(Filter,BaseDir,FileName)
Enumerates recursively all files FileName (absolute file name) in BaseDir directory files that match the corresponding Filter.
- Call and exit should be compatible with:
(source_tree:source_filter/1)Filter is a source file filter
(basic_props:atm/1)BaseDir is an atom.
(basic_props:atm/1)FileName is an atom. - The following properties should hold at call time:
(term_typing:nonvar/1)Filter is currently a term which is not a free variable.
(term_typing:nonvar/1)BaseDir is currently a term which is not a free variable.
(term_typing:var/1)FileName is a free variable.
Usage:copy_file_tree(Filter,SrcDir,DestDir,Perms)
Copy the file tree from SrcDir to DestDir with permissions Perms
Usage:copy_file_tree(Filter,SrcDir,DestDir,Perms,Owner)
Copy the file tree from SrcDir to DestDir with permissions Perms and owner Owner
Usage:clean_file_tree(PrecompLevel,Dir)
Unwind the precompilation level at directory Dir, cleaning the contents recursively (see untainted and cleanable(PrecompLevel) source_filter/1).
Usage:copy_file_or_dir(FileName,DestDir)
Copy FileName file into DestDir directory. The path for DestDir is created if it does not exists. If FileName is a directory, all its contents are copied recursively.
Usage:remove_dir(Dir)
Delete the directory Dir and all its contents recursively.
Usage:remove_file_or_dir(FileName)
Delete FileName. If FileName is a directory, all its contents are deleted recursively.
Usage:delete_glob(Dir,Pattern)
Delete each file in directory Dir (non-recursively) that matches the glob pattern Pattern
Usage:remove_glob(Dir,Pattern)
Like delete_glob/2, but uses remove_dir/1 for each matching directory in Dir.
Basic filters:
- true
- Enumerate all files (recursively).
- nonrec
- Do not recurse into directories (directories will be treated as files).
- untainted
- Enumerate all files, except those that are backups or hold repository metadata.
- srctype(SrcTypes)
- Enumerate source files of any of the specified SrcTypes, a source type or list of source types. A source type (file_srctype/1) is any of typesmodule, package, or include files (for user or included files).
- proj(TreeProj)
- Obtain a tree projection based on marks. Projections allow discarding whole subtrees or individual files. The valid projection names and their marks are:
- compilable
- for each directory, discard the subtree if it contains the NOCOMPILE file, or ignore the files whose name match with any of the patterns listed in the file NOCOMPILEFILES.
- testable
- same with NOTEST and NOTESTFILES (implies compilable).
- distributable
- same with NODISTRIBUTE and NODISTRIBUTEFILES.
- installable
- same with NOINSTALL and NOINSTALLFILES.
- precomp(PrecompLevel)
- Enumerate files for the specified precompilation level PrecompLevel (see precomp_level/1).
- cleanable(PrecompLevel)
- Enumerate the files resulting from compilation of source files that must be cleaned in order reduce the precompilation to PrecompLevel. E.g., PrecompLevel=src cleans all compiler output, PrecompLevel=noa cleans only platform dependant files.
Extended filters:
- compilable_module
- Enumerate Ciao modules that are suitable for automatic compilation during bundle build (in the untainted proj(compilable) projection).
- testable_module
- Enumerate Ciao modules that can be compiled and tested (in the untainted proj(compilable) and proj(testable) projection).
- distributable_precomp(PrecompLevel)
- Enumerate files that can be distributed (in the untainted proj(distributable) projection and for the given precompilation level PrecompLevel).
Additionally, excludes the build directory and some temporary files.
- installable_precomp(PrecompLevel)
- Enumerate (source) files that can be installed (in the untainted proj(installable) projection and for the given precompilation level PrecompLevel).
Additionally, excludes the build directory and some temporary files.
Usage:source_filter(Filter)
Filter is a source file filter
- src
- source files only
- noa
- sources and portable objects
- bin
- sources, portable, and architecture dependant objects (except special third-party files like elisp objects and Java classes)
- full
- like bin, including special third-party files.
Usage:precomp_level(PrecompLevel)
PrecompLevel is a pre-compilation level
file_srctype(module). file_srctype(package). file_srctype(include).
Usage:file_srctype(T)
T is a source file type
Usage:get_file_srctype(FileName,Type)
FileName is a file of type Type (module, package, or include for included or user sources). Fails if the file does not seem Ciao code
- Call and exit should be compatible with:
(basic_props:atm/1)FileName is an atom.
(source_tree:file_srctype/1)Type is a source file type - The following properties should hold at call time:
(term_typing:nonvar/1)FileName is currently a term which is not a free variable.
Known bugs and planned improvements
- (bug) Does not work with CIAOCACHEDIR enabled
- (documentation) Add more examples
- (code) This code and documentation could be greatly improved by language extensions (like hooks definitions)
- (code) Use (;)/2 and (,)/2 for filters
- (code) Use dynamic facts instead of peep_p (the implementation may be simpler and more efficient)
- (code) Merge with bundle_scan.pl? (at least some predicates)
- (performance) Optimize regexp/glob matching
- (performance) We need atom GC
- (feature) Use a single PROPS or META file to store metadata?
- (feature) Extend and document all filter expressions
- (feature) Maybe WalkP can be inserted into DirP and FileP (see appendix)
- (feature) Add filters for system:file_property
- (feature) OR-composition of filters is not supported (we need a more involved algorithm that avoid duplicates)
- (feature) Add a predoc filter? (similar to precomp)
- Implement a copy that preserves original modes, perms, and owner? (like cp -p)