2. Advanced topics

This part of the Scipy lecture notes is dedicated to advanced usage. It strives to educate the proficient Python coder to be an expert and tackles various specific topics.

• 2.1. Advanced Python Constructs
  • 2.1.1. Iterators, generator expressions and generators
    • 2.1.1.1. Iterators
    • 2.1.1.2. Generator expressions
    • 2.1.1.3. Generators
    • 2.1.1.4. Bidirectional communication
    • 2.1.1.5. Chaining generators
  • 2.1.2. Decorators
    • 2.1.2.1. Replacing or tweaking the original object
    • 2.1.2.2. Decorators implemented as classes and as functions
    • 2.1.2.3. Copying the docstring and other attributes of the original function
    • 2.1.2.4. Examples in the standard library
    • 2.1.2.5. Deprecation of functions
    • 2.1.2.6. A while-loop removing decorator
    • 2.1.2.7. A plugin registration system
  • 2.1.3. Context managers
    • 2.1.3.1. Catching exceptions
    • 2.1.3.2. Using generators to define context managers
• 2.2. Advanced Numpy
  • 2.2.1. Life of ndarray
    • 2.2.1.1. It’s...
    • 2.2.1.2. Block of memory
    • 2.2.1.3. Data types
      • 2.2.1.3.1. The descriptor
      • 2.2.1.3.2. Example: reading .wav files
      • 2.2.1.3.3. Casting and re-interpretation/views
        • 2.2.1.3.3.1. Casting
        • 2.2.1.3.3.2. Re-interpretation / viewing
    • 2.2.1.4. Indexing scheme: strides
      • 2.2.1.4.1. Main point
        • 2.2.1.4.1.1. C and Fortran order
        • 2.2.1.4.1.2. Slicing with integers
      • 2.2.1.4.2. Example: fake dimensions with strides
      • 2.2.1.4.3. Broadcasting
      • 2.2.1.4.4. More tricks: diagonals
      • 2.2.1.4.5. CPU cache effects
      • 2.2.1.4.6. Example: inplace operations (caveat emptor)
    • 2.2.1.5. Findings in dissection
  • 2.2.2. Universal functions
    • 2.2.2.1. What they are?
      • 2.2.2.1.1. Parts of an Ufunc
      • 2.2.2.1.2. Making it easier
    • 2.2.2.2. Exercise: building an ufunc from scratch
    • 2.2.2.3. Solution: building an ufunc from scratch
    • 2.2.2.4. Generalized ufuncs
  • 2.2.3. Interoperability features
    • 2.2.3.1. Sharing multidimensional, typed data
    • 2.2.3.2. The old buffer protocol
    • 2.2.3.3. The old buffer protocol
    • 2.2.3.4. Array interface protocol
  • 2.2.4. Array siblings: chararray, maskedarray, matrix
    • 2.2.4.1. chararray: vectorized string operations
    • 2.2.4.2. masked_array missing data
      • 2.2.4.2.1. The mask
      • 2.2.4.2.2. Domain-aware functions
    • 2.2.4.3. recarray: purely convenience
    • 2.2.4.4. matrix: convenience?
  • 2.2.5. Summary
  • 2.2.6. Contributing to Numpy/Scipy
    • 2.2.6.1. Why
    • 2.2.6.2. Reporting bugs
      • 2.2.6.2.1. Good bug report
    • 2.2.6.3. Contributing to documentation
    • 2.2.6.4. Contributing features
    • 2.2.6.5. How to help, in general
• 2.3. Debugging code
  • 2.3.1. Avoiding bugs
    • 2.3.1.1. Coding best practices to avoid getting in trouble
    • 2.3.1.2. pyflakes: fast static analysis
      • 2.3.1.2.1. Running pyflakes on the current edited file
      • 2.3.1.2.2. A type-as-go spell-checker like integration
  • 2.3.2. Debugging workflow
  • 2.3.3. Using the Python debugger
    • 2.3.3.1. Invoking the debugger
      • 2.3.3.1.1. Postmortem
      • 2.3.3.1.2. Step-by-step execution
      • 2.3.3.1.3. Other ways of starting a debugger
    • 2.3.3.2. Debugger commands and interaction
      • 2.3.3.2.1. Getting help when in the debugger
  • 2.3.4. Debugging segmentation faults using gdb
• 2.4. Optimizing code
  • 2.4.1. Optimization workflow
  • 2.4.2. Profiling Python code
    • 2.4.2.1. Timeit
    • 2.4.2.2. Profiler
    • 2.4.2.3. Line-profiler
  • 2.4.3. Making code go faster
    • 2.4.3.1. Algorithmic optimization
      • 2.4.3.1.1. Example of the SVD
  • 2.4.4. Writing faster numerical code
    • 2.4.4.1. Additional Links
• 2.5. Sparse Matrices in SciPy
  • 2.5.1. Introduction
    • 2.5.1.1. Why Sparse Matrices?
    • 2.5.1.2. Sparse Matrices vs. Sparse Matrix Storage Schemes
    • 2.5.1.3. Typical Applications
    • 2.5.1.4. Prerequisites
    • 2.5.1.5. Sparsity Structure Visualization
  • 2.5.2. Storage Schemes
    • 2.5.2.1. Common Methods
    • 2.5.2.2. Sparse Matrix Classes
      • 2.5.2.2.1. Diagonal Format (DIA)
        • 2.5.2.2.1.1. Examples
      • 2.5.2.2.2. List of Lists Format (LIL)
        • 2.5.2.2.2.1. Examples
      • 2.5.2.2.3. Dictionary of Keys Format (DOK)
        • 2.5.2.2.3.1. Examples
      • 2.5.2.2.4. Coordinate Format (COO)
        • 2.5.2.2.4.1. Examples
      • 2.5.2.2.5. Compressed Sparse Row Format (CSR)
        • 2.5.2.2.5.1. Examples
      • 2.5.2.2.6. Compressed Sparse Column Format (CSC)
        • 2.5.2.2.6.1. Examples
      • 2.5.2.2.7. Block Compressed Row Format (BSR)
        • 2.5.2.2.7.1. Examples
    • 2.5.2.3. Summary
  • 2.5.3. Linear System Solvers
    • 2.5.3.1. Sparse Direct Solvers
      • 2.5.3.1.1. Examples
    • 2.5.3.2. Iterative Solvers
      • 2.5.3.2.1. Common Parameters
      • 2.5.3.2.2. LinearOperator Class
      • 2.5.3.2.3. A Few Notes on Preconditioning
    • 2.5.3.3. Eigenvalue Problem Solvers
      • 2.5.3.3.1. The eigen module
  • 2.5.4. Other Interesting Packages
• 2.6. Image manipulation and processing using Numpy and Scipy
  • 2.6.1. Opening and writing to image files
  • 2.6.2. Displaying images
  • 2.6.3. Basic manipulations
    • 2.6.3.1. Statistical information
    • 2.6.3.2. Geometrical transformations
  • 2.6.4. Image filtering
    • 2.6.4.1. Blurring/smoothing
    • 2.6.4.2. Sharpening
    • 2.6.4.3. Denoising
    • 2.6.4.4. Mathematical morphology
  • 2.6.5. Feature extraction
    • 2.6.5.1. Edge detection
    • 2.6.5.2. Segmentation
  • 2.6.6. Measuring objects properties: ndimage.measurements
• 2.7. Mathematical optimization: finding minima of functions
  • 2.7.1. Knowing your problem
    • 2.7.1.1. Convex versus non-convex optimization
    • 2.7.1.2. Smooth and non-smooth problems
    • 2.7.1.3. Noisy versus exact cost functions
    • 2.7.1.4. Constraints
  • 2.7.2. A review of the different optimizers
    • 2.7.2.1. Getting started: 1D optimization
    • 2.7.2.2. Gradient based methods
      • 2.7.2.2.1. Some intuitions about gradient descent
      • 2.7.2.2.2. Conjugate gradient descent
    • 2.7.2.3. Newton and quasi-newton methods
      • 2.7.2.3.1. Newton methods: using the Hessian (2nd differential)
      • 2.7.2.3.2. Quasi-Newton methods: approximating the Hessian on the fly
    • 2.7.2.4. Gradient-less methods
      • 2.7.2.4.1. A shooting method: the Powell algorithm
      • 2.7.2.4.2. Simplex method: the Nelder-Mead
    • 2.7.2.5. Global optimizers
      • 2.7.2.5.1. Brute force: a grid search
  • 2.7.3. Practical guide to optimization with scipy
    • 2.7.3.1. Choosing a method
    • 2.7.3.2. Making your optimizer faster
    • 2.7.3.3. Computing gradients
    • 2.7.3.4. Synthetic exercices
  • 2.7.4. Special case: non-linear least-squares
    • 2.7.4.1. Minimizing the norm of a vector function
    • 2.7.4.2. Curve fitting
  • 2.7.5. Optimization with constraints
    • 2.7.5.1. Box bounds
    • 2.7.5.2. General constraints
• 2.8. Interfacing with C
  • 2.8.1. Introduction
  • 2.8.2. Python-C-Api
    • 2.8.2.1. Example
    • 2.8.2.2. Numpy Support
  • 2.8.3. Ctypes
    • 2.8.3.1. Example
    • 2.8.3.2. Numpy Support
  • 2.8.4. SWIG
    • 2.8.4.1. Example
    • 2.8.4.2. Numpy Support
  • 2.8.5. Cython
    • 2.8.5.1. Example
    • 2.8.5.2. Numpy Support
  • 2.8.6. Summary
  • 2.8.7. Further Reading and References
  • 2.8.8. Exercises
    • 2.8.8.1. Python-C-API
    • 2.8.8.2. Ctypes
    • 2.8.8.3. SWIG
    • 2.8.8.4. Cython