darwin3 |
|
|
|
|||
Warning, /doc/autodiff/autodiff.rst is written in an unsupported language. File is not indexed.
view on githubraw file Latest commit 626c6ac9 on 2025-11-05 20:20:29 UTC1c65381846 Jeff*0001 .. _chap_autodiff: 0002 0003 Automatic Differentiation 0004 ************************* 0005 5f55d7c73d Jeff*0006 Author: Patrick Heimbach 0007 0008 *Automatic differentiation* (AD), also referred to as algorithmic (or, 0009 more loosely, computational) differentiation, involves automatically 0010 deriving code to calculate partial derivatives from an existing fully 0011 non-linear prognostic code (see Griewank and Walther, 2008 :cite:`griewank:08`). 0012 A software 0013 tool is used that parses and transforms source files according to a set 0014 of linguistic and mathematical rules. AD tools are like source-to-source 0015 translators in that they parse a program code as input and produce a new 0016 program code as output (we restrict our discussion to source-to-source 0017 tools, ignoring operator-overloading tools). However, unlike a pure 0018 source-to-source translation, the output program represents a new 0019 algorithm, such as the evaluation of the Jacobian, the Hessian, or 0020 higher derivative operators. In principle, a variety of derived 0021 algorithms can be generated automatically in this way. 0022 0023 MITgcm has been adapted for use with the Tangent linear and Adjoint 0024 Model Compiler (TAMC) and its successor TAF (Transformation of 0025 Algorithms in Fortran), developed by Ralf Giering 0026 (Giering and Kaminski, 1998 :cite:`giering:98`, Giering, 2000 0027 :cite:`giering:00`). The 0028 first application of the adjoint of MITgcm for sensitivity studies was 0029 published by Marotzke et al. (1999) :cite:`maro-eta:99`. 0030 Stammer et al. (1997, 2002) :cite:`stammer:97` :cite:`stammer:02` use MITgcm and its adjoint 0031 for ocean state estimation studies. In the following we shall refer to 0032 TAMC and TAF synonymously, except were explicitly stated otherwise. 0033 0034 As of mid-2007 we are also able to generate fairly efficient adjoint 0035 code of the MITgcm using a new, open-source AD tool, called OpenAD (see 0036 Naumann, 2006 :cite:`naumann:06` and Utke et al., 2008 :cite:`utke:08`). 0037 This enables us for the 0038 first time to compare adjoint models generated from different AD tools, 0039 providing an additional accuracy check, complementary to 0040 finite-difference gradient checks. OpenAD and its application to MITgcm 0041 is described in detail in :numref:`ad_openad`. 0042 0043 The AD tool exploits the chain rule for computing the first derivative 0044 of a function with respect to a set of input variables. Treating a given** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 46.
0045 forward code as a composition of operations – each line representing a 0046 compositional element, the chain rule is rigorously applied to the code, 0047 line by line. The resulting tangent linear or adjoint code, then, may be 0048 thought of as the composition in forward or reverse order, respectively,** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 50.
0049 of the Jacobian matrices of the forward code’s compositional elements. 0050 0051 Some basic algebra 0052 ================== 0053 0054 Let :math:`\cal{M}` be a general nonlinear, model, i.e., a mapping from 0055 the :math:`m`-dimensional space :math:`U \subset \mathbb{R}^m` of input 0056 variables :math:`\vec{u}=(u_1,\ldots,u_m)` (model parameters, initial 0057 conditions, boundary conditions such as forcing functions) to the 0058 :math:`n`-dimensional space :math:`V \subset \mathbb{R}^n` of model output 0059 variable :math:`\vec{v}=(v_1,\ldots,v_n)` (model state, model 0060 diagnostics, objective function, ...) under consideration: 0061 0062 .. math:: 0063 \begin{aligned} 0064 {\cal M} \, : & \, U \,\, \longrightarrow \, V \\ b4daa24319 Shre*0065 ~ & \, \vec{u} \,\, \longmapsto \, \vec{v} \, = \, 5f55d7c73d Jeff*0066 {\cal M}(\vec{u})\end{aligned} 0067 :label: fulloperator b4daa24319 Shre*0068 5f55d7c73d Jeff*0069 The vectors :math:`\vec{u} \in U` and :math:`\vec{v} \in V` may be 0070 represented with respect to some given basis vectors 0071 :math:`{\rm span} (U) = \{ {\vec{e}_i} \}_{i = 1, \ldots , m}` and 0072 :math:`{\rm span} (V) = \{ {\vec{f}_j} \}_{j = 1, \ldots , n}` as 0073 0074 .. math:: 0075 \vec{u} \, = \, \sum_{i=1}^{m} u_i \, {\vec{e}_i}, 0076 \qquad 0077 \vec{v} \, = \, \sum_{j=1}^{n} v_j \, {\vec{f}_j} 0078 0079 Two routes may be followed to determine the sensitivity of the output 0080 variable :math:`\vec{v}` to its input :math:`\vec{u}`. 0081 0082 Forward or direct sensitivity 0083 ----------------------------- 0084 0085 Consider a perturbation to the input variables :math:`\delta \vec{u}` 0086 (typically a single component 0087 :math:`\delta \vec{u} = \delta u_{i} \, {\vec{e}_{i}}`). Their effect on 0088 the output may be obtained via the linear approximation of the model 0089 :math:`{\cal M}` in terms of its Jacobian matrix :math:`M`, evaluated 0090 in the point :math:`u^{(0)}` according to 0091 0092 .. math:: 0093 \delta \vec{v} \, = \, M |_{\vec{u}^{(0)}} \, \delta \vec{u} 0094 :label: tangent_linear 0095 0096 with resulting output perturbation :math:`\delta \vec{v}`. In 0097 components 0098 :math:`M_{j i} \, = \, \partial {\cal M}_{j} / \partial u_{i}`, it 0099 reads 0100 0101 .. math:: b4daa24319 Shre*0102 \delta v_{j} \, = \, \sum_{i} 0103 \left. \frac{\partial {\cal M}_{j}}{\partial u_{i}} \right|_{u^{(0)}} \, 5f55d7c73d Jeff*0104 \delta u_{i} 0105 :label: jacobi_matrix 0106 0107 :eq:`tangent_linear` is the tangent linear model (TLM). In contrast 0108 to the full nonlinear model :math:`{\cal M}`, the operator :math:`M` 0109 is just a matrix which can readily be used to find the forward 0110 sensitivity of :math:`\vec{v}` to perturbations in :math:`u`, but if 0111 there are very many input variables :math:`(\gg O(10^{6})` for 0112 large-scale oceanographic application), it quickly becomes prohibitive 0113 to proceed directly as in :eq:`tangent_linear`, if the impact of each 0114 component :math:`{\bf e_{i}}` is to be assessed. 0115 0116 Reverse or adjoint sensitivity 0117 ------------------------------ 0118 0119 Let us consider the special case of a scalar objective function 0120 :math:`{\cal J}(\vec{v})` of the model output (e.g., the total meridional 0121 heat transport, the total uptake of CO\ :sub:`2` in the Southern Ocean 0122 over a time interval, or a measure of some model-to-data misfit) 0123 0124 .. math:: 0125 \begin{aligned} 0126 \begin{array}{cccccc} b4daa24319 Shre*0127 {\cal J} \, : & U & 0128 \longrightarrow & V & 5f55d7c73d Jeff*0129 \longrightarrow & \mathbb{R} \\ b4daa24319 Shre*0130 ~ & \vec{u} & \longmapsto & \vec{v}={\cal M}(\vec{u}) & 5f55d7c73d Jeff*0131 \longmapsto & {\cal J}(\vec{u}) = {\cal J}({\cal M}(\vec{u})) 0132 \end{array}\end{aligned} 0133 :label: compo 0134 0135 The perturbation of :math:`{\cal J}` around a fixed point 0136 :math:`{\cal J}_0`, 0137 0138 .. math:: {\cal J} \, = \, {\cal J}_0 \, + \, \delta {\cal J} 0139 0140 can be expressed in both bases of :math:`\vec{u}` and 0141 :math:`\vec{v}` with respect to their corresponding inner product 0142 :math:`\left\langle \,\, , \,\, \right\rangle` 0143 0144 .. math:: 0145 \begin{aligned} 0146 {\cal J} & = \, b4daa24319 Shre*0147 {\cal J} |_{\vec{u}^{(0)}} \, + \, 0148 \left\langle \, \nabla _{u}{\cal J}^T |_{\vec{u}^{(0)}} \, , \, \delta \vec{u} \, \right\rangle 5f55d7c73d Jeff*0149 \, + \, O(\delta \vec{u}^2) \\ 0150 ~ & = \, b4daa24319 Shre*0151 {\cal J} |_{\vec{v}^{(0)}} \, + \, 5f55d7c73d Jeff*0152 \left\langle \, \nabla _{v}{\cal J}^T |_{\vec{v}^{(0)}} \, , \, \delta \vec{v} \, \right\rangle 0153 \, + \, O(\delta \vec{v}^2) 0154 \end{aligned} 0155 :label: deljidentity 0156 0157 (note, that the gradient :math:`\nabla f` is a co-vector, therefore 0158 its transpose is required in the above inner product). Then, using the 0159 representation of :math:`\delta {\cal J} = 0160 \left\langle \, \nabla _{v}{\cal J}^T \, , \, \delta \vec{v} \, \right\rangle`, 0161 the definition of an adjoint operator :math:`A^{\ast}` of a given 0162 operator :math:`A`, 0163 0164 .. math:: 0165 \left\langle \, A^{\ast} \vec{x} \, , \, \vec{y} \, \right\rangle = 0166 \left\langle \, \vec{x} \, , \, A \vec{y} \, \right\rangle 0167 0168 which for finite-dimensional vector spaces is just the transpose of 0169 :math:`A`, 0170 0171 .. math:: A^{\ast} \, = \, A^T 0172 0173 and from :eq:`tangent_linear`, :eq:`deljidentity`, we note that** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 175.
0174 (omitting :math:`|`\ ’s): 0175 0176 .. math:: 0177 \delta {\cal J} 0178 \, = \, 0179 \left\langle \, \nabla _{v}{\cal J}^T \, , \, \delta \vec{v} \, \right\rangle 0180 \, = \, 0181 \left\langle \, \nabla _{v}{\cal J}^T \, , \, M \, \delta \vec{u} \, \right\rangle b4daa24319 Shre*0182 \, = \, 0183 \left\langle \, M^T \, \nabla _{v}{\cal J}^T \, , \, 5f55d7c73d Jeff*0184 \delta \vec{u} \, \right\rangle 0185 :label: inner 0186 0187 With the identity :eq:`deljidentity`, we then find that the gradient 0188 :math:`\nabla _{u}{\cal J}` can be readily inferred by invoking the 0189 adjoint :math:`M^{\ast }` of the tangent linear model :math:`M` 0190 0191 .. math:: 0192 \begin{aligned} b4daa24319 Shre*0193 \nabla _{u}{\cal J}^T |_{\vec{u}} & 5f55d7c73d Jeff*0194 = \, M^T |_{\vec{u}} \cdot \nabla _{v}{\cal J}^T |_{\vec{v}} \\ 0195 ~ & = \, M^T |_{\vec{u}} \cdot \delta \vec{v}^{\ast} \\ 0196 ~ & = \, \delta \vec{u}^{\ast} 0197 \end{aligned} 0198 :label: adjoint 0199 0200 :eq:`adjoint` is the adjoint model (ADM), in which :math:`M^T` is the 0201 adjoint (here, the transpose) of the tangent linear operator :math:`M`, 0202 :math:`\,\delta \vec{v}^{\ast}` the adjoint variable of the model state 0203 :math:`\vec{v}`, and :math:`\delta \vec{u}^{\ast}` the adjoint 0204 variable of the control variable :math:`\vec{u}`. 0205 0206 The reverse nature of the adjoint calculation can be readily seen as 0207 follows. Consider a model integration which consists of 0208 :math:`\Lambda` consecutive operations 0209 :math:`{\cal M}_{\Lambda} ( {\cal M}_{\Lambda-1} ( ...... ( {\cal M}_{\lambda} (...... 0210 ( {\cal M}_{1} ( {\cal M}_{0}(\vec{u}) ))))`, where the** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 212.
0211 :math:`{\cal M}`\ ’s could be the elementary steps, i.e., single lines in 0212 the code of the model, or successive time steps of the model 0213 integration, starting at step 0 and moving up to step :math:`\Lambda`, 0214 with intermediate 0215 :math:`{\cal M}_{\lambda} (\vec{u}) = \vec{v}^{(\lambda+1)}` and final 0216 :math:`{\cal M}_{\Lambda} (\vec{u}) = \vec{v}^{(\Lambda+1)} = \vec{v}`. 0217 Let :math:`{\cal J}` be a cost function which explicitly depends on the 0218 final state :math:`\vec{v}` only (this restriction is for clarity 0219 reasons only). :math:`{\cal J}(u)` may be decomposed according to: 0220 0221 .. math:: b4daa24319 Shre*0222 {\cal J}({\cal M}(\vec{u})) \, = \, 0223 {\cal J} ( {\cal M}_{\Lambda} ( {\cal M}_{\Lambda-1} ( 5f55d7c73d Jeff*0224 ...... ( {\cal M}_{\lambda} (...... 0225 ( {\cal M}_{1} ( {\cal M}_{0}(\vec{u}) ))))) 0226 :label: compos 0227 0228 Then, according to the chain rule, the forward calculation reads, in** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 230.
0229 terms of the Jacobi matrices (we’ve omitted the :math:`|`\ ’s which, 0230 nevertheless are important to the aspect of *tangent* linearity; note 0231 also that by definition 0232 :math:`\langle \, \nabla _{v}{\cal J}^T \, , \, \delta \vec{v} \, \rangle 0233 = \nabla_v {\cal J} \cdot \delta \vec{v}` ) 0234 0235 .. math:: 0236 \begin{aligned} 0237 \nabla_v {\cal J} (M(\delta \vec{u})) & = \, 0238 \nabla_v {\cal J} \cdot M_{\Lambda} 0239 \cdot ...... \cdot M_{\lambda} \cdot ...... \cdot 0240 M_{1} \cdot M_{0} \cdot \delta \vec{u} \\ 0241 ~ & = \, \nabla_v {\cal J} \cdot \delta \vec{v} \\ 0242 \end{aligned} 0243 :label: forward 0244 0245 whereas in reverse mode we have 0246 0247 .. math:: 0248 \boxed{ 0249 \begin{aligned} 0250 M^T ( \nabla_v {\cal J}^T) & = \, 0251 M_{0}^T \cdot M_{1}^T b4daa24319 Shre*0252 \cdot ...... \cdot M_{\lambda}^T \cdot ...... \cdot 5f55d7c73d Jeff*0253 M_{\Lambda}^T \cdot \nabla_v {\cal J}^T \\ 0254 ~ & = \, M_{0}^T \cdot M_{1}^T b4daa24319 Shre*0255 \cdot ...... \cdot 5f55d7c73d Jeff*0256 \nabla_{v^{(\lambda)}} {\cal J}^T \\ 0257 ~ & = \, \nabla_u {\cal J}^T 0258 \end{aligned}} 0259 :label: reverse 0260 0261 clearly expressing the reverse nature of the calculation. 0262 :eq:`reverse` is at the heart of automatic adjoint compilers. If the** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 264.
0263 intermediate steps :math:`\lambda` in :eq:`compos` – :eq:`reverse` 0264 represent the model state (forward or adjoint) at each intermediate time 0265 step as noted above, then correspondingly, 0266 :math:`M^T (\delta \vec{v}^{(\lambda) \, \ast}) = 0267 \delta \vec{v}^{(\lambda-1) \, \ast}` for the adjoint variables. It 0268 thus becomes evident that the adjoint calculation also yields the 0269 adjoint of each model state component :math:`\vec{v}^{(\lambda)}` at 0270 each intermediate step :math:`\lambda`, namely 0271 0272 .. math:: 0273 \boxed{ 0274 \begin{aligned} 0275 \nabla_{v^{(\lambda)}} {\cal J}^T |_{\vec{v}^{(\lambda)}} 0276 & = \, b4daa24319 Shre*0277 M_{\lambda}^T |_{\vec{v}^{(\lambda)}} \cdot ...... \cdot 5f55d7c73d Jeff*0278 M_{\Lambda}^T |_{\vec{v}^{(\lambda)}} \cdot \delta \vec{v}^{\ast} \\ 0279 ~ & = \, \delta \vec{v}^{(\lambda) \, \ast} 0280 \end{aligned}} 0281 0282 in close analogy to :eq:`adjoint` we note in passing that the 0283 :math:`\delta \vec{v}^{(\lambda) \, \ast}` are the Lagrange multipliers 0284 of the model equations which determine :math:`\vec{v}^{(\lambda)}`. 0285 0286 In components, :eq:`adjoint` reads as follows. Let 0287 0288 .. math:: 0289 \begin{array}{rclcrcl} 0290 \delta \vec{u} & = & 0291 \left( \delta u_1,\ldots, \delta u_m \right)^T , & \qquad & 0292 \delta \vec{u}^{\ast} \,\, = \,\, \nabla_u {\cal J}^T & = & b4daa24319 Shre*0293 \left( 0294 \frac{\partial {\cal J}}{\partial u_1},\ldots, 5f55d7c73d Jeff*0295 \frac{\partial {\cal J}}{\partial u_m} 0296 \right)^T \\ 0297 \delta \vec{v} & = & 0298 \left( \delta v_1,\ldots, \delta u_n \right)^T , & \qquad & 0299 \delta \vec{v}^{\ast} \,\, = \,\, \nabla_v {\cal J}^T & = & b4daa24319 Shre*0300 \left( 0301 \frac{\partial {\cal J}}{\partial v_1},\ldots, 5f55d7c73d Jeff*0302 \frac{\partial {\cal J}}{\partial v_n} 0303 \right)^T \\ 0304 \end{array} 0305 0306 denote the perturbations in :math:`\vec{u}` and :math:`\vec{v}`, 0307 respectively, and their adjoint variables; further 0308 0309 .. math:: 0310 M \, = \, \left( 0311 \begin{array}{ccc} 0312 \frac{\partial {\cal M}_1}{\partial u_1} & \ldots & 0313 \frac{\partial {\cal M}_1}{\partial u_m} \\ 0314 \vdots & ~ & \vdots \\ 0315 \frac{\partial {\cal M}_n}{\partial u_1} & \ldots & 0316 \frac{\partial {\cal M}_n}{\partial u_m} \\ 0317 \end{array} 0318 \right) 0319 0320 is the Jacobi matrix of :math:`{\cal M}` (an :math:`n \times m` 0321 matrix) such that :math:`\delta \vec{v} = M \cdot \delta \vec{u}`, or 0322 0323 .. math:: b4daa24319 Shre*0324 \delta v_{j} 5f55d7c73d Jeff*0325 \, = \, \sum_{i=1}^m M_{ji} \, \delta u_{i} b4daa24319 Shre*0326 \, = \, \sum_{i=1}^m \, \frac{\partial {\cal M}_{j}}{\partial u_{i}} 5f55d7c73d Jeff*0327 \delta u_{i} 0328 0329 Then :eq:`adjoint` takes the form 0330 0331 .. math:: b4daa24319 Shre*0332 \delta u_{i}^{\ast} 5f55d7c73d Jeff*0333 \, = \, \sum_{j=1}^n M_{ji} \, \delta v_{j}^{\ast} b4daa24319 Shre*0334 \, = \, \sum_{j=1}^n \, \frac{\partial {\cal M}_{j}}{\partial u_{i}} 5f55d7c73d Jeff*0335 \delta v_{j}^{\ast} 0336 0337 or 0338 0339 .. math:: 0340 \left( 0341 \begin{array}{c} 0342 \left. \frac{\partial}{\partial u_1} {\cal J} \right|_{\vec{u}^{(0)}} \\ 0343 \vdots \\ 0344 \left. \frac{\partial}{\partial u_m} {\cal J} \right|_{\vec{u}^{(0)}} \\ 0345 \end{array} 0346 \right) 0347 \, = \, 0348 \left( 0349 \begin{array}{ccc} b4daa24319 Shre*0350 \left. \frac{\partial {\cal M}_1}{\partial u_1} \right|_{\vec{u}^{(0)}} 5f55d7c73d Jeff*0351 & \ldots & 0352 \left. \frac{\partial {\cal M}_n}{\partial u_1} \right|_{\vec{u}^{(0)}} \\ 0353 \vdots & ~ & \vdots \\ b4daa24319 Shre*0354 \left. \frac{\partial {\cal M}_1}{\partial u_m} \right|_{\vec{u}^{(0)}} 5f55d7c73d Jeff*0355 & \ldots & 0356 \left. \frac{\partial {\cal M}_n}{\partial u_m} \right|_{\vec{u}^{(0)}} \\ 0357 \end{array} 0358 \right) 0359 \cdot 0360 \left( 0361 \begin{array}{c} 0362 \left. \frac{\partial}{\partial v_1} {\cal J} \right|_{\vec{v}} \\ 0363 \vdots \\ 0364 \left. \frac{\partial}{\partial v_n} {\cal J} \right|_{\vec{v}} \\ 0365 \end{array} 0366 \right) 0367 0368 Furthermore, the adjoint :math:`\delta v^{(\lambda) \, \ast}` of any 0369 intermediate state :math:`v^{(\lambda)}` may be obtained, using the 0370 intermediate Jacobian (an :math:`n_{\lambda+1} \times n_{\lambda}` 0371 matrix) 0372 0373 .. math:: 0374 M_{\lambda} \, = \, 0375 \left( 0376 \begin{array}{ccc} 0377 \frac{\partial ({\cal M}_{\lambda})_1}{\partial v^{(\lambda)}_1} 0378 & \ldots & 0379 \frac{\partial ({\cal M}_{\lambda})_1}{\partial v^{(\lambda)}_{n_{\lambda}}} \\ 0380 \vdots & ~ & \vdots \\ 0381 \frac{\partial ({\cal M}_{\lambda})_{n_{\lambda+1}}}{\partial v^{(\lambda)}_1} 0382 & \ldots & 0383 \frac{\partial ({\cal M}_{\lambda})_{n_{\lambda+1}}}{\partial v^{(\lambda)}_{n_{\lambda}}} \\ 0384 \end{array} 0385 \right) 0386 0387 and the shorthand notation for the adjoint variables 0388 :math:`\delta v^{(\lambda) \, \ast}_{j} = \frac{\partial}{\partial v^{(\lambda)}_{j}} 0389 {\cal J}^T`, :math:`j = 1, \ldots , n_{\lambda}`, for intermediate 0390 components, yielding 0391 0392 .. math:: 0393 \begin{aligned} 0394 \left( 0395 \begin{array}{c} 0396 \delta v^{(\lambda) \, \ast}_1 \\ 0397 \vdots \\ 0398 \delta v^{(\lambda) \, \ast}_{n_{\lambda}} \\ 0399 \end{array} 0400 \right) 0401 \, = & 0402 \left( 0403 \begin{array}{ccc} 0404 \frac{\partial ({\cal M}_{\lambda})_1}{\partial v^{(\lambda)}_1} 0405 & \ldots \,\, \ldots & 0406 \frac{\partial ({\cal M}_{\lambda})_{n_{\lambda+1}}}{\partial v^{(\lambda)}_1} \\ 0407 \vdots & ~ & \vdots \\ 0408 \frac{\partial ({\cal M}_{\lambda})_1}{\partial v^{(\lambda)}_{n_{\lambda}}} 0409 & \ldots \,\, \ldots & 0410 \frac{\partial ({\cal M}_{\lambda})_{n_{\lambda+1}}}{\partial v^{(\lambda)}_{n_{\lambda}}} \\ 0411 \end{array} 0412 \right) 0413 \cdot 0414 % 0415 \\ ~ & ~ 0416 \\ ~ & 0417 % 0418 \left( 0419 \begin{array}{ccc} 0420 \frac{\partial ({\cal M}_{\lambda+1})_1}{\partial v^{(\lambda+1)}_1} 0421 & \ldots & 0422 \frac{\partial ({\cal M}_{\lambda+1})_{n_{\lambda+2}}}{\partial v^{(\lambda+1)}_1} \\ 0423 \vdots & ~ & \vdots \\ 0424 \vdots & ~ & \vdots \\ 0425 \frac{\partial ({\cal M}_{\lambda+1})_1}{\partial v^{(\lambda+1)}_{n_{\lambda+1}}} 0426 & \ldots & 0427 \frac{\partial ({\cal M}_{\lambda+1})_{n_{\lambda+2}}}{\partial v^{(\lambda+1)}_{n_{\lambda+1}}} \\ 0428 \end{array} 0429 \right) 0430 \cdot \, \ldots \, \cdot 0431 \left( 0432 \begin{array}{c} 0433 \delta v^{\ast}_1 \\ 0434 \vdots \\ 0435 \delta v^{\ast}_{n} \\ 0436 \end{array} 0437 \right) 0438 \end{aligned} 0439 0440 :eq:`forward` and :eq:`reverse` are perhaps clearest in showing the 0441 advantage of the reverse over the forward mode if the gradient 0442 :math:`\nabla _{u}{\cal J}`, i.e., the sensitivity of the cost function 0443 :math:`{\cal J}` with respect to *all* input variables :math:`u` (or 0444 the sensitivity of the cost function with respect to *all* intermediate 0445 states :math:`\vec{v}^{(\lambda)}`) are sought. In order to be able to 0446 solve for each component of the gradient 0447 :math:`\partial {\cal J} / \partial u_{i}` in :eq:`forward` a forward 0448 calculation has to be performed for each component separately, i.e., 0449 :math:`\delta \vec{u} = \delta u_{i} {\vec{e}_{i}}` for the 0450 :math:`i`-th forward calculation. Then, :eq:`forward` represents the 0451 projection of :math:`\nabla_u {\cal J}` onto the :math:`i`-th 0452 component. The full gradient is retrieved from the :math:`m` forward 0453 calculations. In contrast, :eq:`reverse` yields the full gradient 0454 :math:`\nabla _{u}{\cal J}` (and all intermediate gradients 0455 :math:`\nabla _{v^{(\lambda)}}{\cal J}`) within a single reverse 0456 calculation. 0457 0458 Note, that if :math:`{\cal J}` is a vector-valued function of 0459 dimension :math:`l > 1`, :eq:`reverse` has to be modified according 0460 to 0461 0462 .. math:: b4daa24319 Shre*0463 M^T \left( \nabla_v {\cal J}^T \left(\delta \vec{J}\right) \right) 5f55d7c73d Jeff*0464 \, = \, 0465 \nabla_u {\cal J}^T \cdot \delta \vec{J} 0466 0467 where now :math:`\delta \vec{J} \in \mathbb{R}^l` is a vector of 0468 dimension :math:`l`. In this case :math:`l` reverse simulations have 0469 to be performed for each :math:`\delta J_{k}, \,\, k = 1, \ldots, l`. 0470 Then, the reverse mode is more efficient as long as :math:`l < n`, 0471 otherwise the forward mode is preferable. Strictly, the reverse mode is 0472 called adjoint mode only for :math:`l = 1`. 0473 0474 A detailed analysis of the underlying numerical operations shows that 0475 the computation of :math:`\nabla _{u}{\cal J}` in this way requires 0476 about two to five times the computation of the cost function. Alternatively, 0477 the gradient vector could be approximated by finite differences, 0478 requiring :math:`m` computations of the perturbed cost function. 0479 0480 To conclude, we give two examples of commonly used types of cost 0481 functions: 0482 0483 Example 1: :math:`{\cal J} = v_{j} (T)` 0484 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 0485 0486 The cost function consists of the :math:`j`-th component of the model 0487 state :math:`\vec{v}` at time :math:`T`. Then 0488 :math:`\nabla_v {\cal J}^T = {\vec{f}_{j}}` is just the :math:`j`-th 0489 unit vector. The :math:`\nabla_u {\cal J}^T` is the projection of 0490 the adjoint operator onto the :math:`j`-th component 0491 :math:`{\bf f_{j}}`, 0492 0493 .. math:: b4daa24319 Shre*0494 \nabla_u {\cal J}^T 5f55d7c73d Jeff*0495 \, = \, M^T \cdot \nabla_v {\cal J}^T 0496 \, = \, \sum_{i} M^T_{ji} \, {\vec{e}_{i}} 0497 0498 Example 2: :math:`{\cal J} = \langle \, {\cal H}(\vec{v}) - \vec{d} \, , \, {\cal H}(\vec{v}) - \vec{d} \, \rangle` 0499 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 0500 0501 The cost function represents the quadratic model vs. data misfit. 0502 Here, :math:`\vec{d}` is the data vector and :math:`{\cal H}` 0503 represents the operator which maps the model state space onto the data 0504 space. Then, :math:`\nabla_v {\cal J}` takes the form 0505 0506 .. math:: 0507 \begin{aligned} b4daa24319 Shre*0508 \nabla_v {\cal J}^T & = \, 2 \, \, H \cdot 5f55d7c73d Jeff*0509 \left( \, {\cal H}(\vec{v}) - \vec{d} \, \right) \\ 0510 ~ & = \, 2 \sum_{j} \left\{ \sum_k b4daa24319 Shre*0511 \frac{\partial {\cal H}_k}{\partial v_{j}} 5f55d7c73d Jeff*0512 \left( {\cal H}_k (\vec{v}) - d_k \right) 0513 \right\} \, {\vec{f}_{j}} \\ 0514 \end{aligned} 0515 0516 where :math:`H_{kj} = \partial {\cal H}_k / \partial v_{j}` is the 0517 Jacobi matrix of the data projection operator. Thus, the gradient 0518 :math:`\nabla_u {\cal J}` is given by the adjoint operator, driven 0519 by the model vs. data misfit: 0520 0521 .. math:: b4daa24319 Shre*0522 \nabla_u {\cal J}^T \, = \, 2 \, M^T \cdot 5f55d7c73d Jeff*0523 H \cdot \left( {\cal H}(\vec{v}) - \vec{d} \, \right) 0524 d67096e55c Jeff*0525 .. _sec_autodiff_storage_v_recompute: 0526 5f55d7c73d Jeff*0527 Storing vs. recomputation in reverse mode 0528 ----------------------------------------- 0529 0530 We note an important aspect of the forward vs. reverse mode calculation. 0531 Because of the local character of the derivative (a derivative is 0532 defined with respect to a point along the trajectory), the intermediate results 0533 of the model trajectory 0534 :math:`\vec{v}^{(\lambda+1)}={\cal M}_{\lambda}(v^{(\lambda)})` may be 0535 required to evaluate the intermediate Jacobian 0536 :math:`M_{\lambda}|_{\vec{v}^{(\lambda)}} \, \delta \vec{v}^{(\lambda)}`. 0537 This is the case for example for nonlinear expressions (momentum advection, 0538 nonlinear equation of state), and state-dependent conditional statements 0539 (parameterization schemes). In the forward mode, the intermediate 0540 results are required in the same order as computed by the full forward 0541 model :math:`{\cal M}`, but in the reverse mode they are required in the 0542 reverse order. Thus, in the reverse mode the trajectory of the forward 0543 model integration :math:`{\cal M}` has to be stored to be available in 0544 the reverse calculation. Alternatively, the complete model state up to 0545 the point of evaluation has to be recomputed whenever its value is 0546 required. 0547 0548 A method to balance the amount of recomputations vs. storage 0549 requirements is called checkpointing (e.g., Griewank, 1992 :cite:`griewank:92`, 0550 Restrepo et al., 1998 :cite:`restrepo:98`). It is depicted in :numref:`checkpointing` for 0551 a 3-level checkpointing (as an example, we give explicit numbers for a 0552 3-day integration with a 1-hourly timestep in square brackets). 0553 0554 0555 .. figure:: figs/checkpointing.png 0556 :width: 100% 0557 :align: center 0558 :alt: 3-lvl checkpointing schematic figure 0559 :name: checkpointing 0560 0561 Schematic view of intermediate dump and restart for 3-level checkpointing. 0562 0563 - In a first step, the model trajectory is subdivided into 0564 :math:`{n}^{lev3}` subsections [:math:`{n}^{lev3}`\ =3 1-day 0565 intervals], with the label :math:`lev3` for this outermost loop. The 0566 model is then integrated along the full trajectory, and the model 0567 state stored to disk only at every :math:`k_{i}^{lev3}`-th timestep 0568 [i.e. 3 times, at :math:`i = 0,1,2` corresponding to 0569 :math:`k_{i}^{lev3} = 0, 24, 48`]. In addition, the cost function 0570 is computed, if needed. 0571 0572 - In a second step each subsection itself is divided into 0573 :math:`{n}^{lev2}` subsections [:math:`{n}^{lev2}`\ =4 6-hour 0574 intervals per subsection]. The model picks up at the last outermost 0575 dumped state :math:`v_{k_{n}^{lev3}}` and is integrated forward in 0576 time along the last subsection, with the label :math:`lev2` for this 0577 intermediate loop. The model state is now stored to disk at every 0578 :math:`k_{i}^{lev2}`-th timestep [i.e. 4 times, at 0579 :math:`i = 0,1,2,3` corresponding to 0580 :math:`k_{i}^{lev2} = 48, 54, 60, 66`]. 0581 0582 - Finally, the model picks up at the last intermediate dump state 0583 :math:`v_{k_{n}^{lev2}}` and is integrated forward in time along 0584 the last subsection, with the label :math:`lev1` for this 0585 intermediate loop. Within this sub-subsection only, parts of the 0586 model state are stored to memory at every timestep [i.e. every hour 0587 :math:`i=0,...,5` corresponding to 0588 :math:`k_{i}^{lev1} = 66, 67, \ldots, 71`]. The final state 0589 :math:`v_n = v_{k_{n}^{lev1}}` is reached and the model state of 0590 all preceding timesteps along the last innermost subsection are 0591 available, enabling integration backwards in time along the last 0592 subsection. The adjoint can thus be computed along this last 0593 subsection :math:`k_{n}^{lev2}`. 0594 0595 This procedure is repeated consecutively for each previous subsection 0596 :math:`k_{n-1}^{lev2}, \ldots, k_{1}^{lev2}` carrying the adjoint 0597 computation to the initial time of the subsection :math:`k_{n}^{lev3}`. 0598 Then, the procedure is repeated for the previous subsection 0599 :math:`k_{n-1}^{lev3}` carrying the adjoint computation to the initial 0600 time :math:`k_{1}^{lev3}`. 0601 0602 For the full model trajectory of 0603 :math:`n^{lev3} \cdot n^{lev2} \cdot n^{lev1}` timesteps the required 0604 storing of the model state was significantly reduced to 0605 :math:`n^{lev2} + n^{lev3}` to disk and roughly :math:`n^{lev1}` to 0606 memory (i.e., for the 3-day integration with a total of 72 timesteps the 0607 model state was stored 7 times to disk and roughly 6 times to memory). 0608 This saving in memory comes at a cost of a required 3 full forward 0609 integrations of the model (one for each checkpointing level). The 0610 optimal balance of storage vs. recomputation certainly depends on the 0611 computing resources available and may be adjusted by adjusting the 0612 partitioning among the :math:`n^{lev3}, \,\, n^{lev2}, \,\, n^{lev1}`. 0613 d67096e55c Jeff*0614 .. _sec_ad_tlm_and_adm: 0615 5f55d7c73d Jeff*0616 TLM and ADM generation in general 0617 ================================= 0618 0619 In this section we describe in a general fashion the parts of the code 0620 that are relevant for automatic differentiation using the software tool 0621 TAF. Modifications to use OpenAD are described in :numref:`ad_openad`. 0622 b4daa24319 Shre*0623 The basic flow is as follows: 5f55d7c73d Jeff*0624 0625 :: 0626 0627 the_model_main 0628 | 0629 |--- initialise_fixed 0630 | 0631 |--- #ifdef ALLOW_ADJOINT_RUN b4daa24319 Shre*0632 | | 5f55d7c73d Jeff*0633 | |--- ctrl_unpack b4daa24319 Shre*0634 | | 5f55d7c73d Jeff*0635 | |--- adthe_main_loop 0636 | | | 0637 | | |--- initialise_varia 0638 | | |--- ctrl_map_forcing 0639 | | |--- do iloop = 1, nTimeSteps 0640 | | | |--- forward_step 0641 | | | |--- cost_tile 0642 | | | end do 0643 | | |--- cost_final 0644 | | | 0645 | | |--- adcost_final 0646 | | |--- do iloop = nTimeSteps, 1, -1 0647 | | | |--- adcost_tile 0648 | | | |--- adforward_step 0649 | | | end do 0650 | | |--- adctrl_map_forcing 0651 | | |--- adinitialise_varia 0652 | | o 0653 | | 0654 | |--- ctrl_pack 0655 | | 0656 |--- #else 0657 | | 0658 | |--- the_main_loop 0659 | | 0660 | #endif 0661 | 0662 |--- #ifdef ALLOW_GRADIENT_CHECK 0663 | | 0664 | |--- grdchk_main 0665 | o 0666 | #endif 0667 o 0668 0669 0670 If CPP option 0671 :varlink:`ALLOW_AUTODIFF_TAMC` is defined, the driver routine 0672 :filelink:`the_model_main.F <model/src/the_model_main.F>`, 0673 instead of calling :filelink:`the_model_loop.F <model/src/the_main_loop.F>`, invokes the 0674 adjoint of this routine, ``adthe_main_loop.F`` (case 0675 #define :varlink:`ALLOW_ADJOINT_RUN`, or the tangent linear of this routine 0676 ``g_the_main_loop.F`` (case #define :varlink:`ALLOW_TANGENTLINEAR_RUN`), which 0677 are the toplevel routines in terms of automatic differentiation. The 0678 routines ``adthe_main_loop.F`` or ``g_the_main_loop.F`` are generated by 0679 TAF. It contains both the forward integration of the full model, the 0680 cost function calculation, any additional storing that is required for 0681 efficient checkpointing, and the reverse integration of the adjoint 0682 model. 0683 0684 [DESCRIBE IN A SEPARATE SECTION THE WORKING OF THE TLM] 0685 0686 The above structure of ``adthe_main_loop.F`` has been 0687 strongly simplified to focus on the essentials; in particular, no 0688 checkpointing procedures are shown here. Prior to the call of 0689 ``adthe_main_loop.F``, the routine :filelink:`ctrl_unpack.F <pkg/ctrl/ctrl_unpack.F>` 0690 is invoked to unpack the 0691 control vector or initialize the control variables. Following the call 0692 of ``adthe_main_loop.F``, the routine :filelink:`ctrl_pack.F <pkg/ctrl/ctrl_pack.F>` 0693 is invoked to pack the 0694 control vector (cf. :numref:`the_ctrl_vars`). If gradient checks are to 0695 be performed, the option #define :varlink:`ALLOW_GRDCHK` is chosen. In this case 0696 the driver routine :filelink:`grdchk_main.F <pkg/grdchk/grdchk_main.F>` 0697 is called after the gradient has been 0698 computed via the adjoint (cf. :numref:`ad_gradient_check`). 0699 0700 General setup 0701 ------------- 0702 0703 In order to configure AD-related setups the following packages need to 0704 be enabled: 0705 0706 - :filelink:`pkg/autodiff` 0707 - :filelink:`pkg/ctrl` 0708 - :filelink:`pkg/cost` 0709 - :filelink:`pkg/grdchk` 0710 0711 The packages are enabled by adding them to your experiment-specific d8c5b89513 Ivan*0712 configuration file ``packages.conf`` (see :numref:`using_packages`). 5f55d7c73d Jeff*0713 0714 The following AD-specific CPP option files need to be customized: 0715 d8c5b89513 Ivan*0716 - :filelink:`AUTODIFF_OPTIONS.h <pkg/autodiff/AUTODIFF_OPTIONS.h>` This header 0717 file collects CPP options for :filelink:`pkg/autodiff`, :filelink:`pkg/cost`, 0718 :filelink:`pkg/ctrl` as well as AD-unrelated options for the external forcing 0719 package :filelink:`pkg/exf`. 0720 0721 - :filelink:`COST_OPTIONS.h <pkg/cost/COST_OPTIONS.h>` In this header file, 0722 options for different cost functions are set. 0723 0724 - :filelink:`CTRL_OPTIONS.h <pkg/ctrl/CTRL_OPTIONS.h>` In this header file the 0725 control variables are enabled and options for writing and reading the control 0726 vector are set 5f55d7c73d Jeff*0727 0728 - :filelink:`tamc.h <pkg/autodiff/tamc.h>` 0729 This header configures the splitting of the time stepping loop 0730 with respect to the 3-level checkpointing (see section ???). 0731 31584ea246 Jeff*0732 .. _building_adcode_using_taf: 0733 5f55d7c73d Jeff*0734 Building the AD code using TAF 0735 ------------------------------ 0736 0737 The build process of an AD code is very similar to building the forward 0738 model. However, depending on which AD code one wishes to generate, and 0739 on which AD tool is available (TAF or TAMC), the following make targets 0740 are available: 0741 0742 +------------------+------------------------+----------------------------------------------------------------------------------+ 0743 | *AD-target* | *output* | *description* | 0744 +==================+========================+==================================================================================+ 0745 | «MODE»«TOOL»only | «MODE»_«TOOL»_output.f | generates code for «MODE» using «TOOL» | 0746 +------------------+------------------------+----------------------------------------------------------------------------------+ 0747 | | | no make dependencies on .F .h | 0748 +------------------+------------------------+----------------------------------------------------------------------------------+ 0749 | | | useful for compiling on remote platforms | 0750 +------------------+------------------------+----------------------------------------------------------------------------------+ 0751 | «MODE»«TOOL» | «MODE»_«TOOL»_output.f | generates code for «MODE» using «TOOL» | 0752 +------------------+------------------------+----------------------------------------------------------------------------------+ 0753 | | | includes make dependencies on .F .h | 0754 +------------------+------------------------+----------------------------------------------------------------------------------+ 0755 | | | i.e. input for «TOOL» may be re-generated | 0756 +------------------+------------------------+----------------------------------------------------------------------------------+ 0757 | «MODE»all | mitgcmuv\_«MODE» | generates code for «MODE» using «TOOL» | 0758 +------------------+------------------------+----------------------------------------------------------------------------------+ 0759 | | | and compiles all code | 0760 +------------------+------------------------+----------------------------------------------------------------------------------+ 0761 | | | (use of TAF is set as default) | 0762 +------------------+------------------------+----------------------------------------------------------------------------------+ 0763 0764 Here, the following placeholders are used: 0765 0766 - «TOOL» 0767 0768 - TAF 0769 0770 - TAMC 0771 0772 - «MODE» 0773 0774 - ad generates the adjoint model (ADM) 0775 0776 - ftl generates the tangent linear model (TLM) 0777 0778 - svd generates both ADM and TLM for 0779 singular value decomposition (SVD) type calculations 0780 0781 For example, to generate the adjoint model using TAF after routines (``.F``) 0782 or headers (``.h``) have been modified, but without compilation, 0783 type ``make adtaf``; or, to generate the tangent linear model using TAMC without 0784 re-generating the input code, type ``make ftltamconly``. 0785 0786 A typical full build process to generate the ADM via TAF would look like 0787 follows: 0788 0789 :: 0790 0791 % mkdir build 0792 % cd build d8c5b89513 Ivan*0793 % ../../../tools/genmake2 -mods=../code_ad [ -nocat4ad ] 5f55d7c73d Jeff*0794 % make depend 0795 % make adall 0796 0797 0798 The AD build process in detail 0799 ------------------------------ 0800 0801 The ``make «MODE»all`` target consists of the following procedures: 0802 0803 #. A header file ``AD_CONFIG.h`` is generated which contains a CPP option 0804 on which code ought to be generated. Depending on the ``make`` target, 0805 the contents is one of the following: 0806 0807 - #define :varlink:`ALLOW_ADJOINT_RUN` 0808 0809 - #define :varlink:`ALLOW_TANGENTLINEAR_RUN` 0810 d8c5b89513 Ivan*0811 #. If `` -nocat4ad`` is not specified, a single file ``«MODE»_input_code.f`` is 0812 concatenated consisting of all ``.f`` files that are part of the list 0813 ``AD_FILES`` and all ``.flow`` files that are part of the list 0814 ``AD_FLOW_FILES``. 5f55d7c73d Jeff*0815 0816 #. The AD tool is invoked with the ``«MODE»_«TOOL»_FLAGS``. The default AD tool d8c5b89513 Ivan*0817 flags in :filelink:`genmake2 <tools/genmake2>` can be overwritten by a 0818 :filelink:`tools/adjoint_options` file (similar to the platform-specific 0819 :filelink:`tools/build_options`, see :numref:`genmake2_optfiles`). The AD 0820 tool writes the resulting AD code into the file ``«MODE»_input_code_ad.f``. 5f55d7c73d Jeff*0821 d8c5b89513 Ivan*0822 #. A short sed script :filelink:`tools/adjoint_sed <tools/adjoint_sed>` is 0823 applied to ``«MODE»_input_code_ad.f`` to reinstate :varlink:`myThid` into 0824 the CALL argument list of active file I/O. The result is written to file 0825 ``«MODE»_«TOOL»_output.f``. 0826 0827 #. If the `` -nocat4ad`` option is specified, the concatenation of all ``.f`` 0828 files is skipped and instead all necessary files are sent to TAF and for 0829 each file an AD-file is returned. 5f55d7c73d Jeff*0830 0831 #. All routines are compiled and an executable is generated. 0832 f955de4ba2 Jean*0833 The list ``AD_FILES`` and ``*_ad_diff.list`` files 0834 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 5f55d7c73d Jeff*0835 0836 Not all routines are presented to the AD tool. Routines typically hidden 0837 are diagnostics routines which do not influence the cost function, but 0838 may create artificial flow dependencies such as I/O of active variables. 0839 f955de4ba2 Jean*0840 :filelink:`genmake2 <tools/genmake2>` generates a list (or variable) ``AD_FILES`` 0841 that contains all routines that are shown to the AD tool. 0842 This list is put together from all files with suffix ``_ad_diff.list`` 0843 that :filelink:`genmake2 <tools/genmake2>` finds in its search directories. 0844 The list file for the core MITgcm routines is :filelink:`model/src/model_ad_diff.list`. 0845 Note that no wrapper routine is shown to TAF. These are either not visible at 0846 all to the AD code, or hand-written AD code is available (see next section). 5f55d7c73d Jeff*0847 0848 Each package directory contains its package-specific list file 0849 ``«PKG»_ad_diff.list``. For example, :filelink:`pkg/ptracers` contains the file f955de4ba2 Jean*0850 :filelink:`ptracers_ad_diff.list <pkg/ptracers/ptracers_ad_diff.list>`. 5f55d7c73d Jeff*0851 Thus, enabling a package will automatically 0852 extend the ``AD_FILES`` list of :filelink:`genmake2 <tools/genmake2>` to incorporate the 0853 package-specific routines. Note that you will need to regenerate the 0854 makefile if you enable a package (e.g., by adding it to ``packages.conf``) 0855 and a ``Makefile`` already exists. 0856 0857 The list ``AD_FLOW_FILES`` and ``.flow`` files 0858 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 0859 0860 TAMC and TAF can evaluate user-specified directives that start with a 0861 specific syntax (``CADJ``, ``C$TAF``, ``!$TAF``). The main categories of directives 0862 are ``STORE`` directives and ``FLOW`` directives. Here, we are concerned with 0863 flow directives, store directives are treated elsewhere. 0864 0865 Flow directives enable the AD tool to evaluate how it should treat** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 867.
0866 routines that are ’hidden’ by the user, i.e. routines which are not 0867 contained in the ``AD_FILES`` list (see previous section), but which 0868 are called in part of the code that the AD tool does see. The flow 0869 directive tell the AD tool: 0870 0871 - which subroutine arguments are input/output 0872 0873 - which subroutine arguments are active 0874 0875 - which subroutine arguments are required to compute the cost 0876 0877 - which subroutine arguments are dependent 0878 0879 The syntax for the flow directives can be found in the AD tool manuals. 0880 f955de4ba2 Jean*0881 :filelink:`genmake2 <tools/genmake2>` generates a list (or variable) ``AD_FLOW_FILES`` 0882 that contains all files with suffix ``.flow`` that it finds in its search 5f55d7c73d Jeff*0883 directories. The flow directives for the core MITgcm routines of 0884 :filelink:`eesupp/src/` and :filelink:`model/src/` reside in :filelink:`pkg/autodiff/`. This directory also 0885 contains hand-written adjoint code for the MITgcm WRAPPER (:numref:`wrapper`). 0886 0887 Flow directives for package-specific routines are contained in the f955de4ba2 Jean*0888 corresponding package directories, generally in a file ``«PKG»_ad.flow``, e.g., 5f55d7c73d Jeff*0889 ptracers-specific directives are in :filelink:`ptracers_ad.flow <pkg/ptracers/ptracers_ad.flow>`. 0890 0891 Store directives for 3-level checkpointing 0892 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 0893 0894 The storing that is required at each period of the 3-level checkpointing 0895 is controlled by three top-level headers. 0896 0897 :: 0898 0899 do ilev_3 = 1, nchklev_3 0900 # include ``checkpoint_lev3.h'' 0901 do ilev_2 = 1, nchklev_2 0902 # include ``checkpoint_lev2.h'' 0903 do ilev_1 = 1, nchklev_1 0904 # include ``checkpoint_lev1.h'' 0905 0906 ... 0907 0908 end do 0909 end do 0910 end do 0911 0912 All files ``checkpoint_lev?.h`` are contained in directory :filelink:`pkg/autodiff/`. 0913 31584ea246 Jeff*0914 .. _adoptfile: 0915 5f55d7c73d Jeff*0916 Changing the default AD tool flags: ad_options files 0917 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 0918 0919 Hand-written adjoint code 0920 ~~~~~~~~~~~~~~~~~~~~~~~~~ 0921 d67096e55c Jeff*0922 .. _pkg_cost_description: 0923 5f55d7c73d Jeff*0924 The cost function (dependent variable) 0925 -------------------------------------- 0926 0927 The cost function :math:`{\cal J}` is referred to as the *dependent 0928 variable*. It is a function of the input variables :math:`\vec{u}` via 0929 the composition 0930 :math:`{\cal J}(\vec{u}) \, = \, {\cal J}(M(\vec{u}))`. The input are 0931 referred to as the *independent variables* or *control variables*. All 0932 aspects relevant to the treatment of the cost function 0933 :math:`{\cal J}` (parameter setting, initialization, accumulation, 0934 final evaluation), are controlled by the package :filelink:`pkg/cost`. The aspects 0935 relevant to the treatment of the independent variables are controlled by 0936 the package :filelink:`pkg/ctrl` and will be treated in the next section. 0937 0938 :: 0939 0940 the_model_main 0941 | 0942 |-- initialise_fixed 0943 | | 0944 | |-- packages_readparms 0945 | | 0946 | |-- cost_readparms 0947 | o 0948 | 0949 |-- the_main_loop 0950 ... | 0951 |-- initialise_varia 0952 | | 0953 | |-- packages_init_variables 0954 | | 0955 | |-- cost_init 0956 | o 0957 | 0958 |-- do iloop = 1,nTimeSteps 0959 | |-- forward_step 0960 | |-- cost_tile 0961 | | | 0962 | | |-- cost_tracer 0963 | end do 0964 | 0965 |-- cost_final 0966 o 0967 0968 Enabling the package 0969 ~~~~~~~~~~~~~~~~~~~~ 0970 d8c5b89513 Ivan*0971 :filelink:`pkg/cost <pkg/cost>` is enabled by adding the line ``cost`` to your 0972 file ``packages.conf`` (see :numref:`using_packages`). 5f55d7c73d Jeff*0973 d8c5b89513 Ivan*0974 In general the following packages ought to be enabled simultaneously: 0975 :filelink:`pkg/autodiff <pkg/autodiff>`, :filelink:`pkg/ctrl <pkg/ctrl>`, and 0976 :filelink:`pkg/cost`. The basic CPP option to enable the cost function is 0977 :varlink:`ALLOW_COST`. Each specific cost function contribution has its own 0978 option. For the present example the option is :varlink:`ALLOW_COST_TRACER`. All 0979 cost-specific options are set in :filelink:`COST_OPTIONS.h 0980 <pkg/ctrl/COST_OPTIONS.h>`. Since the cost function is usually used in 5f55d7c73d Jeff*0981 conjunction with automatic differentiation, the CPP option d8c5b89513 Ivan*0982 :varlink:`ALLOW_AUTODIFF_TAMC` (file :filelink:`AUTODIFF_OPTIONS.h 0983 <pkg/autodiff/AUTODIFF_OPTIONS.h>`) should be defined. 5f55d7c73d Jeff*0984 0985 Initialization 0986 ~~~~~~~~~~~~~~ 0987 0988 The initialization of :filelink:`pkg/cost` is readily enabled as soon as 0989 the CPP option :varlink:`ALLOW_COST` is defined. 0990 0991 - The S/R :filelink:`cost_readparms.F </pkg/cost/cost_readparms.F>` 0992 reads runtime flags and parameters from file ``data.cost``. 0993 For the present example the only relevant parameter read is 0994 :varlink:`mult_tracer`. This multiplier enables different cost function 0995 contributions to be switched on (``= 1.``) or off (``= 0.``) at runtime. 0996 For more complex cost functions which involve model vs. data 0997 misfits, the corresponding data filenames and data specifications 0998 (start date and time, period, ...) are read in this S/R. 0999 1000 - The S/R :filelink:`cost_init_varia.F </pkg/cost/cost_init_varia.F>` 1001 initializes the different cost function contributions. The 1002 contribution for the present example is :varlink:`objf_tracer` which is 1003 defined on each tile (bi,bj). 1004 1005 Accumulation 1006 ~~~~~~~~~~~~ 1007** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 1009.
1008 The ’driver’ routine :filelink:`cost_tile.F </pkg/cost/cost_tile.F>` 1009 is called at the end of each time** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 1011.
1010 step. Within this ’driver’ routine, S/R are called for each of the 1011 chosen cost function contributions. In the present example 1012 (:varlink:`ALLOW_COST_TRACER`), S/R :filelink:`cost_tracer.F </pkg/cost/cost_tracer.F>` is called. It accumulates 1013 :varlink:`objf_tracer` according to eqn. (ref:ask-the-author). 1014 d67096e55c Jeff*1015 .. _sec_ad_finalize_contribtuions: 1016 5f55d7c73d Jeff*1017 Finalize all contributions 1018 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 1019 1020 At the end of the forward integration S/R :filelink:`cost_final.F </pkg/cost/cost_final.F>` is called. It 1021 accumulates the total cost function :varlink:`fc` from each contribution and 1022 sums over all tiles: 1023 1024 .. math:: b4daa24319 Shre*1025 {\cal J} \, = \, 1026 {\rm fc} \, = \, 5f55d7c73d Jeff*1027 {\rm mult\_tracer} \sum_{\text{global sum}} \sum_{bi,\,bj}^{nSx,\,nSy} 1028 {\rm objf\_tracer}(bi,bj) \, + \, ... 1029** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 1031.
1030 The total cost function :varlink:`fc` will be the ’dependent’ variable in the 1031 argument list for TAF, i.e., 1032 1033 :: 1034 1035 taf -output 'fc' ... 1036 1037 :: 1038 1039 ************* 1040 the_main_loop 1041 ************* 1042 | 1043 |--- initialise_varia 1044 | | 1045 | ... 1046 | |--- packages_init_varia 1047 | | | 1048 | | ... 1049 | | |--- #ifdef ALLOW_ADJOINT_RUN 1050 | | | call ctrl_map_ini 1051 | | | call cost_ini 1052 | | | #endif 1053 | | ... 1054 | | o 1055 | ... 1056 | o 1057 ... 1058 |--- #ifdef ALLOW_ADJOINT_RUN 1059 | call ctrl_map_forcing 1060 | #endif 1061 ... 1062 |--- #ifdef ALLOW_TAMC_CHECKPOINTING 1063 do ilev_3 = 1,nchklev_3 1064 | do ilev_2 = 1,nchklev_2 1065 | do ilev_1 = 1,nchklev_1 1066 | iloop = (ilev_3-1)*nchklev_2*nchklev_1 + 1067 | (ilev_2-1)*nchklev_1 + ilev_1 1068 | #else 1069 | do iloop = 1, nTimeSteps 1070 | #endif 1071 | | 1072 | |--- call forward_step 1073 | | 1074 | |--- #ifdef ALLOW_COST 1075 | | call cost_tile 1076 | | #endif 1077 | | 1078 | | enddo 1079 | o 1080 | 1081 |--- #ifdef ALLOW_COST 1082 | call cost_final 1083 | #endif 1084 o 1085 1086 .. _the_ctrl_vars: 1087 1088 The control variables (independent variables) 1089 --------------------------------------------- 1090 1091 The control variables are a subset of the model input (initial 1092 conditions, boundary conditions, model parameters). Here we identify 1093 them with the variable :math:`\vec{u}`. All intermediate variables 1094 whose derivative with respect to control variables do not vanish are called 1095 active variables. All subroutines whose derivative with respect to the control** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 1097.
1096 variables don’t vanish are called active routines. Read and write 1097 operations from and to file can be viewed as variable assignments. 1098 Therefore, files to which active variables are written and from which 1099 active variables are read are called active files. All aspects relevant 1100 to the treatment of the control variables (parameter setting, 1101 initialization, perturbation) are controlled by the package :filelink:`pkg/ctrl`. 1102 1103 :: 1104 1105 the_model_main 1106 | 1107 |-- initialise_fixed 1108 | | 1109 | |-- packages_readparms 1110 | | 1111 | |-- cost_readparms 1112 | o 1113 | 1114 |-- the_main_loop 1115 ... | 1116 |-- initialise_varia 1117 | | 1118 | |-- packages_init_variables 1119 | | 1120 | |-- cost_init 1121 | o 1122 | 1123 |-- do iloop = 1,nTimeSteps 1124 | |-- forward_step 1125 | |-- cost_tile 1126 | | | 1127 | | |-- cost_tracer 1128 | end do 1129 | 1130 |-- cost_final 1131 o 1132 1133 1134 :filelink:`genmake2 <tools/genmake2>` and CPP options 1135 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1136 d8c5b89513 Ivan*1137 Package :filelink:`pkg/ctrl` is enabled by adding the line ``ctrl`` to your 1138 file ``packages.conf``. Each control variable is enabled via its own CPP 1139 option in :filelink:`CTRL_OPTIONS.h <pkg/ctrl/CTRL_OPTIONS.h>`. 5f55d7c73d Jeff*1140 1141 Initialization 1142 ~~~~~~~~~~~~~~ 1143 1144 - The S/R :filelink:`ctrl_readparms.F </pkg/ctrl/ctrl_readparms.F>` 1145 reads runtime flags and parameters from file ``data.ctrl``. 1146 For the present example the file contains the file names of each 1147 control variable that is used. In addition, the number of wet 1148 points for each control variable and the net dimension of the space 1149 of control variables (counting wet points only) :varlink:`nvarlength` is 1150 determined. Masks for wet points for each tile (bi,bj) and 1151 vertical layer k are generated for the three relevant 1152 categories on the C-grid: :varlink:`nWetCtile` for tracer fields, 1153 :varlink:`nWetWtile` for zonal velocity fields, :varlink:`nWetStile` for 1154 meridional velocity fields. 1155 1156 - Two important issues related to the handling of the control 1157 variables in MITgcm need to be addressed. First, in order to save 1158 memory, the control variable arrays are not kept in memory, but 1159 rather read from file and added to the initial fields during the 1160 model initialization phase. Similarly, the corresponding adjoint 1161 fields which represent the gradient of the cost function with respect to the 1162 control variables are written to file at the end of the adjoint 1163 integration. Second, in addition to the files holding the 2-D 1164 and 3-D control variables and the corresponding cost gradients, 1165 a 1-D control vector and gradient vector are written to file. 1166 They contain only the wet points of the control variables and the 1167 corresponding gradient. This leads to a significant data 1168 compression. Furthermore, an option is available 1169 (:varlink:`ALLOW_NONDIMENSIONAL_CONTROL_IO`) to non-dimensionalize the 1170 control and gradient vector, which otherwise would contain 1171 different pieces of different magnitudes and units. Finally, the 1172 control and gradient vector can be passed to a minimization routine 1173 if an update of the control variables is sought as part of a 1174 minimization exercise. 1175 1176 The files holding fields and vectors of the control variables and 1177 gradient are generated and initialized in S/R :filelink:`ctrl_unpack.F </pkg/ctrl/ctrl_unpack.F>`. 1178 1179 Perturbation of the independent variables 1180 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1181 1182 The dependency flow for differentiation with respect to the controls starts with 1183 adding a perturbation onto the input variable, thus defining the 1184 independent or control variables for TAF. Three types of controls may be 1185 considered: 1186 1187 - Consider as an example the initial tracer distribution :varlink:`pTracer` as 1188 control variable. After :varlink:`pTracer` has been initialized in 1189 :filelink:`ptracers_init_varia.F <pkg/ptracers/ptracers_init_varia.F>` 1190 (dynamical variables such as temperature and salinity are 1191 initialized in :filelink:`ini_fields.F <>model/src/ini_fields.F>`), a perturbation anomaly is added to 1192 the field in S/R :filelink:`ctrl_map_ini.F </pkg/ctrl/ctrl_map_ini.F>`: 1193 1194 .. math:: 1195 \begin{aligned} 1196 u & = \, u_{[0]} \, + \, \Delta u \\ 1197 {\bf tr1}(...) & = \, {\bf tr1_{ini}}(...) \, + \, {\bf xx\_tr1}(...) 1198 \end{aligned} 1199 :label: perturb 1200 1201 :varlink:`xx_tr1` is a 3-D global array holding the perturbation. In 1202 the case of a simple sensitivity study this array is identical to** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 1204.
1203 zero. However, it’s specification is essential in the context of 1204 automatic differentiation since TAF treats the corresponding line 1205 in the code symbolically when determining the differentiation chain 1206 and its origin. Thus, the variable names are part of the argument 1207 list when calling TAF: 1208 1209 :: 1210 1211 taf -input 'xx_tr1 ...' ... 1212 1213 Now, as mentioned above, MITgcm avoids maintaining an array for each 1214 control variable by reading the perturbation to a temporary array 1215 from file. To ensure the symbolic link to be recognized by TAF, a** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 1217.
1216 scalar dummy variable ``xx_tr1_dummy`` is introduced and an ’active** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 1218.
1217 read’ routine of the adjoint support package :filelink:`pkg/autodiff` is 1218 invoked. The read-procedure is tagged with the variable 1219 ``xx_tr1_dummy`` enabling TAF to recognize the initialization of 1220 the perturbation. The modified call of TAF thus reads 1221 1222 :: 1223 1224 taf -input 'xx_tr1_dummy ...' ... 1225 1226 and the modified operation (to perturb) in the code takes on the 1227 form 1228 1229 :: 1230 b4daa24319 Shre*1231 call active_read_xyz( 5f55d7c73d Jeff*1232 & ..., tmpfld3d, ..., xx_tr1_dummy, ... ) 1233 1234 tr1(...) = tr1(...) + tmpfld3d(...) 1235 1236 Note that reading an active variable corresponds to a variable 1237 assignment. Its derivative corresponds to a write statement of the** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 1239.
1238 adjoint variable, followed by a reset. The ’active file’ routines 1239 have been designed to support active read and corresponding adjoint 1240 active write operations (and vice versa). 1241 1242 - The handling of boundary values as control variables proceeds 1243 exactly analogous to the initial values with the symbolic 1244 perturbation taking place in S/R 1245 :filelink:`ctrl_map_forcing.F </pkg/ctrl/ctrl_map_forcing.F>`. 1246 Note however 1247 an important difference: Since the boundary values are time 1248 dependent with a new forcing field applied at each time step, the 1249 general problem may be thought of as a new control variable at each 1250 time step (or, if the perturbation is averaged over a certain 1251 period, at each :math:`N` timesteps), i.e., 1252 1253 .. math:: 1254 u_{\rm forcing} \, = \, 1255 \{ \, u_{\rm forcing} ( t_n ) \, \}_{ 1256 n \, = \, 1, \ldots , {\rm nTimeSteps} } 1257 1258 In the current example an equilibrium state is considered, and 1259 only an initial perturbation to surface forcing is applied with 1260 respect to the equilibrium state. A time dependent treatment of the 1261 surface forcing is implemented in the ECCO environment, involving 1262 the calendar (:filelink:`pkg/cal`) and external forcing (:filelink:`pkg/exf`) packages. 1263 1264 - This routine is not yet implemented, but would proceed proceed 1265 along the same lines as the initial value sensitivity. The mixing 1266 parameters :varlink:`diffkr` and :varlink:`kapgm` are currently added as controls 1267 in :filelink:`ctrl_map_ini.F </pkg/ctrl/ctrl_map_ini.F>`. 1268 b4daa24319 Shre*1269 .. _sec_autodiff_output_adj_vars: d67096e55c Jeff*1270 5f55d7c73d Jeff*1271 Output of adjoint variables and gradient 1272 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1273 1274 Several ways exist to generate output of adjoint fields. 1275 1276 - In :filelink:`ctrl_map_ini.F </pkg/ctrl/ctrl_map_ini.F>`, :filelink:`ctrl_map_forcing.F </pkg/ctrl/ctrl_map_forcing.F>`: 1277 1278 - The control variable fields ``xx\_«...»``: before the forward integration, the control variables are read 1279 from file ``«xx\_ ...»`` and added to the model field. 1280 1281 - The adjoint variable fields ``adxx\_«...»``, i.e., the gradient 1282 :math:`\nabla _{u}{\cal J}` for each control variable: 1283 after the adjoint integration the corresponding adjoint 1284 variables are written to ``adxx\_«...»``. 1285 b4daa24319 Shre*1286 - In :filelink:`ctrl_unpack.F </pkg/ctrl/ctrl_unpack.F>`, :filelink:`ctrl_pack.F </pkg/ctrl/ctrl_pack.F>`: 5f55d7c73d Jeff*1287 1288 - The control vector ``vector_ctrl``: 1289 at the very beginning of the model initialization, the updated 1290 compressed control vector is read (or initialized) and 1291 distributed to 2-D and 3-D control variable fields. 1292 1293 - The gradient vector ``vector_grad``: 1294 at the very end of the adjoint integration, the 2-D and 1295 3-D adjoint variables are read, compressed to a single vector 1296 and written to file. 1297 1298 - In addition to writing the gradient at the end of the 1299 forward/adjoint integration, many more adjoint variables of the 1300 model state at intermediate times can be written using S/R 1301 :filelink:`addummy_in_stepping.F </pkg/autodiff/addummy_in_stepping.F>`. 1302 The procedure is 1303 enabled using via the CPP-option :varlink:`ALLOW_AUTODIFF_MONITOR` (file d8c5b89513 Ivan*1304 :filelink:`AUTODIFF_OPTIONS.h <pkg/autodiff/AUTODIFF_OPTIONS.h>`). 5f55d7c73d Jeff*1305 To be part of the adjoint code, the 1306 corresponding S/R :filelink:`dummy_in_stepping.F <pkg/autodiff/dummy_in_stepping.F>` 1307 has to be called in the 1308 forward model (S/R :filelink:`the_main_loop.F <model/src/the_main_loop.F>`) at the appropriate place. The 1309 adjoint common blocks are extracted from the adjoint code via the 1310 header file :filelink:`adcommon.h </pkg/autodiff/adcommon.h>`. 1311 1312 :filelink:`dummy_in_stepping.F <pkg/autodiff/dummy_in_stepping.F>` is essentially empty, the corresponding adjoint 1313 routine is hand-written rather than generated automatically. 1314 Appropriate flow directives 1315 (:filelink:`dummy_in_stepping.flow <pkg/autodiff/dummy_in_stepping.flow>`) 1316 ensure that 1317 TAMC does not automatically generate :filelink:`addummy_in_stepping.F <pkg/autodiff/addummy_in_stepping.F>` by 1318 trying to differentiate :filelink:`dummy_in_stepping.F <pkg/autodiff/dummy_in_stepping.F>`, but instead refers to 1319 the hand-written routine. 1320 1321 :filelink:`dummy_in_stepping.F <pkg/autodiff/dummy_in_stepping.F>` is called in the forward code at the beginning 1322 of each timestep, before the call to :filelink:`model/src/dynamics.F`, thus ensuring that 1323 :filelink:`addummy_in_stepping.F <pkg/autodiff/addummy_in_stepping.F>` is called at the end of each timestep in the 1324 adjoint calculation, after the call to :filelink:`addummy_in_dynamics.F <pkg/autodiff/addummy_in_dynamics.F>`. 1325 1326 :filelink:`addummy_in_stepping.F <pkg/autodiff/addummy_in_stepping.F>` 1327 includes the header files :filelink:`adcommon.h </pkg/autodiff/adcommon.h>`. This 1328 header file is also hand-written. It contains the common blocks 1329 :varlink:`addynvars_r`, :varlink:`addynvars_cd`, :varlink:`addynvars_diffkr`, 1330 :varlink:`addynvars_kapgm`, :varlink:`adtr1_r`, :varlink:`adffields`, which have 1331 been extracted from the adjoint code to enable access to the adjoint 1332 variables. 1333 1334 **WARNING:** If the structure of the common blocks :varlink:`dynvars_r`, 1335 :varlink:`dynvars_cd`, etc., changes similar changes will occur in the 1336 adjoint common blocks. Therefore, consistency between the b4daa24319 Shre*1337 TAMC-generated common blocks and those in 5f55d7c73d Jeff*1338 :filelink:`adcommon.h </pkg/autodiff/adcommon.h>` have to be 1339 checked. 1340 1341 Control variable handling for optimization applications 1342 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1343 1344 In optimization mode the cost function :math:`{\cal J}(u)` is sought 1345 to be minimized with respect to a set of control variables 1346 :math:`\delta {\cal J} \, = \, 0`, in an iterative manner. The 1347 gradient :math:`\nabla _{u}{\cal J} |_{u_{[k]}}` together with the 1348 value of the cost function itself :math:`{\cal J}(u_{[k]})` at 1349 iteration step :math:`k` serve as input to a minimization routine b4daa24319 Shre*1350 (e.g. quasi-Newton method, conjugate gradient, ... (Gilbert and Lemaréchal, 1989 5f55d7c73d Jeff*1351 :cite:`gil-lem:89`) to compute an update in the control 1352 variable for iteration step :math:`k+1`: 1353 1354 .. math:: 1355 u_{[k+1]} \, = \, u_{[0]} \, + \, \Delta u_{[k+1]} 1356 \quad \mbox{satisfying} \quad 1357 {\cal J} \left( u_{[k+1]} \right) \, < \, {\cal J} \left( u_{[k]} \right) 1358 1359 :math:`u_{[k+1]}` then serves as input for a forward/adjoint run to 1360 determine :math:`{\cal J}` and :math:`\nabla _{u}{\cal J}` at 1361 iteration step :math:`k+1`. :numref:`forward-adj_flow` sketches the flow 1362 between forward/adjoint model and the minimization routine. 1363 1364 .. figure:: figs/forward-adj_flow.* 1365 :width: 100% 1366 :align: center 1367 :alt: flow between forward/adjoint model and the minimization 1368 :name: forward-adj_flow 1369 1370 Flow between the forward/adjoint model and the minimization routine. 1371 b4daa24319 Shre*1372 The routines :filelink:`ctrl_unpack.F </pkg/ctrl/ctrl_unpack.F>` and 5f55d7c73d Jeff*1373 :filelink:`ctrl_pack.F </pkg/ctrl/ctrl_pack.F>` provide the link between 1374 the model and the minimization routine. As described in Section 1375 ref:ask-the-author the :filelink:`ctrl_unpack.F </pkg/ctrl/ctrl_unpack.F>` 1376 and :filelink:`ctrl_pack.F </pkg/ctrl/ctrl_pack.F>` routines read and write 1377 control and gradient vectors which are compressed to contain only wet 1378 points, in addition to the full 2-D and 3-D fields. The 1379 corresponding I/O flow is shown in :numref:`forward-adj_io`: 1380 1381 .. figure:: figs/forward-adj_io.* 1382 :width: 100% 1383 :align: center 1384 :alt: forward/adjoint model I/O 1385 :name: forward-adj_io 1386 1387 Flow chart showing I/O in the forward/adjoint model. 1388 1389 d8c5b89513 Ivan*1390 :filelink:`ctrl_unpack.F </pkg/ctrl/ctrl_unpack.F>` reads the updated control 1391 vector ``vector_ctrl_<k>``. It distributes the different control variables to 1392 2-D and 3-D files ``xx_«...»<k>``. At the start of the forward integration the 1393 control variables are read from ``xx_«...»<k>`` and added to the field. 1394 Correspondingly, at the end of the adjoint integration the adjoint fields are 1395 written to ``adxx_«...»<k>``, again via the active file routines. Finally, 1396 :filelink:`ctrl_pack.F </pkg/ctrl/ctrl_pack.F>` collects all adjoint files and 1397 writes them to the compressed vector file ``vector_grad_<k>``. 5f55d7c73d Jeff*1398 1399 1400 .. _ad_gradient_check: 1401 1402 The gradient check package 1403 ========================== 1404 1405 An indispensable test to validate the gradient computed via the adjoint 1406 is a comparison against finite difference gradients. The gradient check 1407 package :filelink:`pkg/grdchk` enables such tests in a straightforward and easy 1408 manner. The driver routine :filelink:`grdchk_main.F <pkg/grdchk/grdchk_main.F>` is called from 1409 :filelink:`the_model_main.F <model/src/the_model_main.F>` after 1410 the gradient has been computed via the adjoint 1411 model (cf. flow chart ???). 1412 1413 The gradient check proceeds as follows: The :math:`i-`\ th component of 1414 the gradient :math:`(\nabla _{u}{\cal J}^T)_i` is compared with the 1415 following finite-difference gradient: 1416 1417 .. math:: 1418 \left(\nabla _{u}{\cal J}^T \right)_i \quad \text{ vs. } \quad 1419 \frac{\partial {\cal J}}{\partial u_i} \, = \, 1420 \frac{ {\cal J}(u_i + \epsilon) - {\cal J}(u_i)}{\epsilon} 1421 1422 A gradient check at point :math:`u_i` may generally considered to be 1423 successful if the deviation of the ratio between the adjoint and the 1424 finite difference gradient from unity is less than 1 percent, 1425 1426 .. math:: b4daa24319 Shre*1427 1 \, - \, 5f55d7c73d Jeff*1428 \frac{({\rm grad}{\cal J})_i (\text{adjoint})} 1429 {({\rm grad}{\cal J})_i (\text{finite difference})} \, < 1 \% 1430 1431 Code description 1432 ---------------- 1433 1434 1435 Code configuration 1436 ------------------ 1437 1438 The relevant CPP precompile options are set in the following files: 1439 1440 - :filelink:`CPP_OPTIONS.h <model/inc/CPP_OPTIONS.h>` 1441 - Together with the flag :varlink:`ALLOW_ADJOINT_RUN`, define the flag :varlink:`ALLOW_GRADIENT_CHECK`. 1442 1443 The relevant runtime flags are set in the files: 1444 1445 - ``data.pkg`` 1446 - Set :varlink:`useGrdchk` ``= .TRUE.`` 1447 1448 - ``data.grdchk`` 1449 1450 - :varlink:`grdchk_eps` 1451 1452 - :varlink:`nbeg` 1453 1454 - :varlink:`nstep` 1455 1456 - :varlink:`nend` 1457 1458 - :varlink:`grdchkvarindex` 1459 1460 :: 1461 1462 the_model_main 1463 | 1464 |-- ctrl_unpack 1465 |-- adthe_main_loop - unperturbed cost function and 1466 |-- ctrl_pack adjoint gradient are computed here 1467 | 1468 |-- grdchk_main 1469 | 1470 |-- grdchk_init 1471 |-- do icomp=... - loop over control vector elements 1472 | 1473 |-- grdchk_loc - determine location of icomp on grid 1474 | 1475 |-- grdchk_getxx - get control vector component from file 1476 | perturb it and write back to file b4daa24319 Shre*1477 |-- grdchk_getadxx - get gradient component calculated 5f55d7c73d Jeff*1478 | via adjoint 1479 |-- the_main_loop - forward run and cost evaluation 1480 | with perturbed control vector element 1481 |-- calculate ratio of adj. vs. finite difference gradient 1482 | 1483 |-- grdchk_setxx - Reset control vector element 1484 | 1485 |-- grdchk_print - print results 1486 d67096e55c Jeff*1487 .. _sec_autodiff_diva: 5f55d7c73d Jeff*1488** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 1490.
1489 Adjoint dump & restart – divided adjoint (DIVA) 1490 =============================================== 1491 d8c5b89513 Ivan*1492 Authors: Patrick Heimbach & Geoffrey Gebbie, 07-Mar-2003 5f55d7c73d Jeff*1493 1494 ***NOTE:THIS SECTION IS SUBJECT TO CHANGE. IT REFERS TO TAF-1.4.26.** 1495 d8c5b89513 Ivan*1496 Old TAF versions are incomplete and have problems with both TAF options 1497 ``-pure`` and ``-mpi``. At the time of the latest update, the current version 1498 of TAF is 6.1.5 5f55d7c73d Jeff*1499 1500 Introduction 1501 ------------ 1502 1503 Most high performance computing (HPC) centers require the use of batch 1504 jobs for code execution. Limits in maximum available CPU time and memory 1505 may prevent the adjoint code execution from fitting into any of the 1506 available queues. This presents a serious limit for large scale / long 1507 time adjoint ocean and climate model integrations. The MITgcm itself 1508 enables the split of the total model integration into sub-intervals 1509 through standard dump/restart of/from the full model state. For a 1510 similar procedure to run in reverse mode, the adjoint model requires, in 1511 addition to the model state, the adjoint model state, i.e., all variables 1512 with derivative information which are needed in an adjoint restart. This** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 1514.
1513 adjoint dump & restart is also termed ’divided adjoint (DIVA)’. 1514 1515 For this to work in conjunction with automatic differentiation, an AD 1516 tool needs to perform the following tasks: 1517 1518 #. identify an adjoint state, i.e., those sensitivities whose 1519 accumulation is interrupted by a dump/restart and which influence the 1520 outcome of the gradient. Ideally, this state consists of 1521 1522 - the adjoint of the model state, 1523 1524 - the adjoint of other intermediate results (such as control 1525 variables, cost function contributions, etc.) 1526 1527 - bookkeeping indices (such as loop indices, etc.) 1528 1529 #. generate code for storing and reading adjoint state variables 1530 1531 #. generate code for bookkeeping , i.e., maintaining a file with index 1532 information 1533 1534 #. generate a suitable adjoint loop to propagate adjoint values for 1535 dump/restart with a minimum overhead of adjoint intermediate values. 1536 1537 TAF (but not TAMC!) generates adjoint code which performs the above 1538 specified tasks. It is closely tied to the adjoint multi-level 1539 checkpointing. The adjoint state is dumped (and restarted) at each step 1540 of the outermost checkpointing level and adjoint integration is 1541 performed over one outermost checkpointing interval. Prior to the 1542 adjoint computations, a full forward sweep is performed to generate the 1543 outermost (forward state) tapes and to calculate the cost function. In 1544 the current implementation, the forward sweep is immediately followed by 1545 the first adjoint leg. Thus, in theory, the following steps are 1546 performed (automatically) 1547 d8c5b89513 Ivan*1548 - **1st model call:** 1549 This is the case if file ``costfinal`` does *not* exist. S/R 5f55d7c73d Jeff*1550 ``mdthe_main_loop.f`` (generated by TAF) is called. 1551 1552 #. calculate forward trajectory and dump model state after each 1553 outermost checkpointing interval to files ``tapelev3`` 1554 1555 #. calculate cost function ``fc`` and write it to file ``costfinal`` 1556 1557 - **2nd and all remaining model calls:** 1558 This is the case if file costfinal *does* exist. S/R 1559 ``adthe_main_loop.f`` (generated by TAF) is called. 1560 1561 #. (forward run and cost function call is avoided since all values 1562 are known) 1563 1564 - if 1st adjoint leg: 1565 create index file ``divided.ctrl`` which contains info on current 1566 checkpointing index :math:`ilev3` 1567 1568 - if not :math:`i`-th adjoint leg: 1569 adjoint picks up at :math:`ilev3 = nlev3-i+1` and runs to 1570 :math:`nlev3 - i` 1571 1572 #. perform adjoint leg from :math:`nlev3-i+1` to :math:`nlev3 - i` 1573 1574 #. dump adjoint state to file ``snapshot`` 1575 1576 #. dump index file ``divided.ctrl`` for next adjoint leg 1577 1578 #. in the last step the gradient is written. 1579 1580 A few modifications were performed in the forward code, obvious ones 1581 such as adding the corresponding TAF-directive at the appropriate place, 1582 and less obvious ones (avoid some re-initializations, when in an 1583 intermediate adjoint integration interval). 1584 1585 [For TAF-1.4.20 a number of hand-modifications were necessary to 1586 compensate for TAF bugs. Since we refer to TAF-1.4.26 onwards, these 1587 modifications are not documented here]. 1588 d8c5b89513 Ivan*1589 .. _diva_recipe: 5f55d7c73d Jeff*1590 d8c5b89513 Ivan*1591 Recipe for divided adjoint code generation 1592 ------------------------------------------ 5f55d7c73d Jeff*1593 d8c5b89513 Ivan*1594 Verification experiment :filelink:`lab_sea <verification/lab_sea>` tests the 1595 divided adjoint and serves as an example of how to configure the code. 5f55d7c73d Jeff*1596 d8c5b89513 Ivan*1597 #. define ``USE_DIVA=1``, either as an environment variable (e.g., in bash: 1598 ``export USE_DIVA=1``), in a ``genmake_local`` file in the ``build`` 1599 directory, or in your build options file. This will instruct 1600 :filelink:`genmake2 <tools/genmake2>` to generate TAF options (``-pure``) 1601 for divided adjoint generation. 5f55d7c73d Jeff*1602 d8c5b89513 Ivan*1603 #. In a local copy of :filelink:`AUTODIFF_OPTIONS.h 1604 <pkg/autodiff/AUTODIFF_OPTIONS.h>` set: 5f55d7c73d Jeff*1605 d8c5b89513 Ivan*1606 - #define :varlink:`ALLOW_DIVIDED_ADJOINT` 5f55d7c73d Jeff*1607 d8c5b89513 Ivan*1608 to enable code for divided adjoint. 5f55d7c73d Jeff*1609 d8c5b89513 Ivan*1610 #. If using MPI, make sure that the paths to mpi-header files, such as 1611 ``mpif.h``, are know to :filelink:`genmake2 <tools/genmake2>` (as usual, via 1612 the build options file, see also :numref:`diva_mpi`). 5f55d7c73d Jeff*1613 d8c5b89513 Ivan*1614 #. Run the usual sequence for generating the Makefile and the AD-code. 5f55d7c73d Jeff*1615 d8c5b89513 Ivan*1616 :: 5f55d7c73d Jeff*1617 f955de4ba2 Jean*1618 ../../../tools/genmake2 -mods=../code_ad -nocat4ad [ other options ] d8c5b89513 Ivan*1619 make depend 1620 make adtaf 5f55d7c73d Jeff*1621 d8c5b89513 Ivan*1622 the ``-nocat4ad`` option is not necessary, but will generate individual 1623 AD-files for each forward file sent to TAF. The adjoint code now contains 1624 subroutines (in ``the_main_loop_ad.f``): 5f55d7c73d Jeff*1625 d8c5b89513 Ivan*1626 - ``adthe_main_loop_ad``: 1627 Is responsible for the forward trajectory, storing of outermost 1628 checkpoint levels to file, computation of cost function, and 1629 storing of cost function to file (1st step). 5f55d7c73d Jeff*1630 d8c5b89513 Ivan*1631 - ``adthe_main_loop``: 1632 Is responsible for computing one adjoint leg, dump adjoint state 1633 to file and write index info to file (2nd and consecutive 1634 steps). 5f55d7c73d Jeff*1635 d8c5b89513 Ivan*1636 Then compile with ``make adall`` (the ``make adtaf`` step is not necessary 1637 unless you want to inspect the TAF-generated code before compiling). 5f55d7c73d Jeff*1638 d8c5b89513 Ivan*1639 .. _diva_mpi: 5f55d7c73d Jeff*1640 d8c5b89513 Ivan*1641 Special considerations for multi processor (MPI) runs 1642 ----------------------------------------------------- 5f55d7c73d Jeff*1643 d8c5b89513 Ivan*1644 On the machine where you execute the code (most likely not the machine where 1645 you run TAF) find the includes directory for MPI containing ``mpif.h``. Either 1646 copy ``mpif.h`` to the machine where you preprocess the code (generate the 1647 ``.f`` files) before TAF-ing, or add the path to the includes directory to your 1648 :filelink:`genmake2 <tools/genmake2>` platform setup. TAF needs some MPI 1649 parameter settings (essentially ``mpi_comm_world`` and ``mpi_integer``) to 1650 incorporate those in the adjoint code. The ``-mpi`` will be added to the TAF 1651 argument list automatically. 5f55d7c73d Jeff*1652 1653 .. _ad_openad: 1654 1655 Adjoint code generation using OpenAD 1656 ==================================== 1657 1658 Authors: Jean Utke, Patrick Heimbach and Chris Hill 1659 1660 Introduction 1661 ------------ 1662 1663 The development of OpenAD was initiated as part of the ACTS (Adjoint 1664 Compiler Technology & Standards) project funded by the NSF Information 1665 Technology Research (ITR) program. The main goals for OpenAD initially 1666 defined for the ACTS project are: 1667 1668 #. develop a flexible, modular, open source tool that can generate 1669 adjoint codes of numerical simulation programs, 1670 1671 #. establish a platform for easy implementation and testing of source 1672 transformation algorithms via a language-independent abstract 1673 intermediate representation, 1674 1675 #. support for source code written in C and Fortan, and 1676 1677 #. generate efficient tangent linear and adjoint for the MIT general 1678 circulation model. 1679** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 1681.
1680 OpenAD’s homepage is at http://www-unix.mcs.anl.gov/OpenAD. A 1681 development WIKI is at** Warning **
Wide character in print at /usr/local/share/lxr/source line 1030, <$git> line 1683.
1682 http://wiki.mcs.anl.gov/OpenAD/index.php/Main_Page. From the WIKI’s 1683 main page, click on `Handling GCM <https://wiki.mcs.anl.gov/OpenAD/index.php/Handling_GCM>`_ 1684 for various aspects pertaining to 1685 differentiating the MITgcm with OpenAD. 1686 1687 Downloading and installing OpenAD 1688 --------------------------------- 1689 1690 The OpenAD webpage has a detailed description on how to download and 1691 build OpenAD. From its homepage, please click on 1692 `Binaries <http://www.mcs.anl.gov/OpenAD/binaries.shtml>`_. You may either download pre-built binaries 1693 for quick trial, or follow the detailed build process described at 1694 http://www.mcs.anl.gov/OpenAD/access.shtml. 1695 1696 Building MITgcm adjoint with OpenAD 1697 ----------------------------------- 1698 1699 **17-January-2008** 1700 1701 OpenAD was successfully built on head node of ``itrda.acesgrid.org``, 1702 for following system: 1703 1704 :: 1705 1706 > uname -a 1707 Linux itrda 2.6.22.2-42.fc6 #1 SMP Wed Aug 15 12:34:26 EDT 2007 i686 i686 i386 GNU/Linux 1708 b4daa24319 Shre*1709 > cat /proc/version 1710 Linux version 2.6.22.2-42.fc6 (brewbuilder@hs20-bc2-4.build.redhat.com) 5f55d7c73d Jeff*1711 (gcc version 4.1.2 20070626 (Red Hat 4.1.2-13)) #1 SMP Wed Aug 15 12:34:26 EDT 2007 1712 1713 > module load ifc/9.1.036 icc/9.1.042 1714 1715 Head of MITgcm branch (``checkpoint59m`` with some modifications) was used for 1716 building adjoint code. Following routing needed special care (revert 1717 to revision 1.1): http://wwwcvs.mitgcm.org/viewvc/MITgcm/MITgcm_contrib/heimbach/OpenAD/OAD_support/active_module.f90?hideattic=0&view=markup. 1718 9d0c386f0c dngo*1719 Building the MITgcm adjoint using an OpenAD Singularity container 1720 ----------------------------------------------------------------- 1721 1722 The MITgcm adjoint can also be built using a Singularity container. You will 1723 need `Singularity <https://singularity.hpcng.org/>`_, version 3.X. A container 1724 with OpenAD can be downloaded from the Sylabs Cloud: [#thanks-Dan]_ 1725 1726 :: 1727 1728 singularity pull library://jahn/default/openad:latest 1729 1730 To use it, supply the path to the downloaded container to genmake2, 1731 1732 :: 1733 1734 ../../../tools/genmake2 -oad -oadsingularity /path/to/openad_latest.sif ... 1735 make adAll 1736 1737 If your build directory is on a remotely mounted file system (mounted at 1738 /mountpoint), you may have to add an option for mounting it in the container: 1739 1740 :: 1741 1742 ../../../tools/genmake2 -oad -oadsngl "-B /mountpoint /path/to/openad_latest.sif" ... 1743 1744 The ``-oadsingularity`` option is also supported by testreport, 1745 :numref:`testreport_utility`. Note that the path to the container has to be 1746 either absolute or relative to the build directory. 1747 b4daa24319 Shre*1748 .. _ad_tapenade: 1749 1750 Adjoint code generation using Tapenade 1751 ====================================== 1752 01ab5437df Shre*1753 Please refer to Gaikwad et al. (2024) :cite:`gaikwad:24` for more details and a comparative analysis with TAF. Recently, introduction of the profiling capabilities in Tapenade have resulted in substantial insights and speedups for the Tapenade-generated adjoint, see Hascoet et al. (2024) :cite:`hascoet:24`. 1754 1755 Feel free to reach out if you wish to use Tapenade and need help! 1756 1757 Authors: Shreyas Sunil Gaikwad, Sri Hari Krishna Naryanan, Laurent Hascoet, Patrick b4daa24319 Shre*1758 Heimbach 1759 1760 Introduction 1761 ------------ 1762 1763 TAPENADE is an open-source Automatic Differentiation Engine developed at INRIA 1764 Sophia-Antipolis by the Tropics then Ecuador teams. TAPENADE can be utilized as 1765 a server (JAVA servlet), which runs at INRIA Sophia-Antipolis. The current 1766 address of this TAPENADE server is `here 1767 <http://www-tapenade.inria.fr:8080/tapenade/index.jsp>`_. TAPENADE can also be 1768 downloaded and installed locally as a set of JAVA classes (JAR archive). In 1769 that case it is run by a simple command line, which can be included into a 1770 Makefile. It also provides you with a user-interface to visualize the results 1771 in a HTML browser. 1772 1773 Downloading and installing Tapenade 1774 ----------------------------------- 1775 1776 While the MITgcm source files are prepared to generate adjoint sensitivities, 1777 they will not be able to do so without an operable installation of 1778 Tapenade. Fortunately the Tapenade installation procedure is straight forward. 1779 1780 We detail the instructions here, but the latest instructions can always be 1781 found `here 1782 <https://tapenade.gitlabpages.inria.fr/tapenade/distrib/README.html>`__. 1783 1784 Prerequisites for Linux or Mac OS 1785 --------------------------------- 1786 1787 Before installing Tapenade, you must check that an up-to-date Java Runtime 1788 Environment is installed. Tapenade will not run with older Java Runtime 1789 Environment. 1790 1791 Steps for Mac OS 1792 ---------------- 1793 626c6ac944 Jean*1794 Tapenade 3.16 distribution does not contain a fortranParser executable 1795 for MacOS. You need docker on your Mac to run the Tapenade 1796 distribution with Fortran programs with a docker image from `here 1797 <https://gitlab.inria.fr/tapenade/tapenade>`__. Details on how to 1798 build your own fortranParser is `here 1799 <https://tapenade.gitlabpages.inria.fr/tapenade/docs/html/src/frontf/README.html?highlight=mac>`__. 1800 You may also build Tapenade on your Mac from the `gitlab repository b4daa24319 Shre*1801 <https://tapenade.gitlabpages.inria.fr/tapenade/docs/html/distrib/README.html>`__. 1802 626c6ac944 Jean*1803 Running a docker image requires absolute paths, e.g., to 1804 :filelink:`tools/TAP_support/flow_tap <tools/TAP_support/flow_tap>`. 1805 To make it work, 1806 1807 1. use the option ``-rootdir`` at the :filelink:`genmake2 1808 <tools/genmake2>` step, or alternatively export environment 1809 variable ``MITGCM_ROOTDIR``, to specify the absolute path to your 1810 MITgcm directory (see also :numref:`command_line_options`). 1811 1812 2. bind mount the absolute path in the docker command as a volume by putting 1813 :: 1814 1815 BASEDIR="$(cd "$(dirname "$0")" && cd ../ && pwd)" 1816 TAPENADECMD="docker container run --rm -u $(stat -f '%u:%g' ./) \ 1817 -v \${PWD}:\${PWD} -v ${BASEDIR}:${BASEDIR} -w \${PWD} \ 1818 registry.gitlab.inria.fr/tapenade/tapenade" 1819 1820 in your build-options or in a ``genmake_local`` file 1821 (:numref:`genmake2_desc`). ``BASENAME`` should expand to your 1822 root directory (check ``TAPENADECMD`` in ``Makefile``). 1823 1824 In order to run :filelink:`./testreport -tap $moreoption 1825 <verification/testreport>` in :filelink:`verification <verification>`, 1826 the root directory can be passed to :filelink:`genmake2 1827 <tools/genmake2>` via ``export MITGCM_ROOTDIR=$BASEDIR`` or setting it 1828 in your built-options or ``genmake_local`` file. b4daa24319 Shre*1829 1830 Steps for Linux 1831 --------------- 1832 1833 1. Read `the Tapenade license. <https://tapenade.gitlabpages.inria.fr/userdoc/build/html/LICENSE.html>`__ 1834 1835 2. Download `tapenade_3.16.tar 1836 <https://tapenade.gitlabpages.inria.fr/tapenade/distrib/tapenade_3.16.tar>`__ 1837 into your chosen installation directory *install_dir*. 1838 1839 3. Go to your chosen installation directory *install_dir*, and extract Tapenade 1840 from the tar file : 1841 1842 :: 1843 1844 % tar xvfz tapenade_3.16.tar 1845 1846 4. On Linux, depending on your distribution, Tapenade may require you to set 1847 the shell variable ``JAVA_HOME`` to your java installation directory. It is 1848 often ``JAVA_HOME=/usr/java/default``. You might also need to modify the 1849 ``PATH`` by adding the bin directory from the Tapenade installation. An 1850 example can be found :ref:`here <tapenade_bashrc_snippet>`. 1851 1852 Prerequisites for Windows 1853 ------------------------- 1854 1855 Before installing Tapenade, you must check that an up-to-date Java Runtime 1856 Environment is installed. Tapenade will not run with older Java Runtime 1857 Environment. The Fortran parser of Tapenade uses `cygwin 1858 <https://www.cygwin.com/>`__. 1859 1860 Steps for Windows 1861 ----------------- 1862 1863 1. Read `the Tapenade license. <https://tapenade.gitlabpages.inria.fr/userdoc/build/html/LICENSE.html>`__ 1864 1865 2. Download `tapenade_3.16.zip 1866 <https://tapenade.gitlabpages.inria.fr/tapenade/distrib/tapenade_3.16.zip>`__ 1867 into your chosen installation directory *install_dir*. 1868 1869 3. Go to your chosen installation directory *install_dir*, and extract Tapenade 1870 from the zip file. 1871 1872 4. Save a copy of the ``install_dir\tapenade_3.16\bin\tapenade.bat`` file and 1873 modify ``install_dir\tapenade_3.16\bin\tapenade.bat`` according to your 1874 installation parameters: 1875 1876 replace ``TAPENADE_HOME=..`` by ``TAPENADE_HOME="install_dir"\tapenade_3.16`` 1877 replace ``JAVA_HOME="C:\Progra~1\Java\jdkXXXX"`` by your current java directory 1878 replace ``BROWSER="C:\Program Files\Internet Explorer\iexplore.exe"`` by your 1879 current browser. 1880 1881 .. _tapenade_bashrc_snippet: 1882 1883 **NOTE**: Every time you wish to use the AD capability with Tapenade, you must re-source the environment. We recommend that this be done automatically in your bash or c-shell profile upon login. An example of an addition to a ``.bashrc`` file from a Linux server is given below. Luckily, shell variable ``JAVA_HOME`` was not required to be explicitly set for this particular Linux distribution, but might be necessary for some other distributions. 1884 1885 :: 1886 1887 ##set some env variables for tapenade 1888 1889 export TAPENADE_HOME="/home/shreyas/tapenade_3.16" 1890 export PATH="$PATH:$TAPENADE_HOME/bin" 1891 1892 ##Modules 1893 1894 module use /share/modulefiles/ 1895 module load java/jdk/16.0.1 # Java required by Tapenade 1896 1897 1898 You should now have a working copy of Tapenade. 1899 1900 For more information on the tapenade command and its arguments, type : 1901 1902 :: 1903 1904 tapenade -? 1905 1906 Prerequisites for Tapenade setup 1907 -------------------------------- 1908 1909 The ``packages.conf`` file should include both the ``adjoint`` and ``tapenade`` 1910 packages. Note that ``mnc`` and ``ecco`` packages are not yet compatible with 1911 Tapenade. The users are referred to the ``code_tap`` directories in the various 1912 verification experiments for reference. 1913 1914 **Pro tip**: ``diff -qr dir1 dir2`` can help you see all the differences in the files of two directories. 1915 1916 ``autodiff`` is not completely untangled from the Tapenade setup yet. In 1917 ``code_tap/AUTODIFF_OPTIONS.h``, the only flag that can be defined safely is 1918 ``ALLOW_AUTODIFF_MONITOR``. 1919 1920 Rest of the setup remains unchanged. 1921 1922 1923 Building MITgcm TLM with Tapenade 1924 --------------------------------- 1925 1926 The setup remains similar to how one sets up the TLM with TAF. A typical flow 1927 will look as follows - 1928 1929 :: 1930 1931 ### Assuming $PWD is the build subdirectory 1932 ### Clean stuff 1933 make CLEAN 1934 1935 ### Use your own optfile 1936 ../../../tools/genmake2 -tap -of ../../../tools/build_options/linux_amd64_ifort -mods ../code_tap 1937 make depend 1938 1939 ### Differentiate code to generate TLM code using Tapenade 1940 ### Creates executable mitgcmuv_tap_tlm 1941 make -j 8 tap_tlm 1942 1943 ### Rest of the setup is standard 1944 cd ../run 1945 rm -r * 1946 ln -s ../input_tap/* . 1947 ../input_tap/prepare_run 1948 ln -s ../build/mitgcmuv_tap_tlm . 1949 ./mitgcmuv_tap_tlm > output_tap_tlm.txt 2>&1 1950 1951 Building MITgcm adjoint with Tapenade 1952 ------------------------------------- 1953 1954 The setup remains similar to how one sets up the adjoint with TAF. A typical 1955 flow will look as follows - 1956 1957 :: 1958 1959 ### Assuming $PWD is the build subdirectory 1960 ### Clean stuff 1961 make CLEAN 1962 1963 ### Use your own optfile 1964 ../../../tools/genmake2 -tap -of ../../../tools/build_options/linux_amd64_ifort -mods ../code_tap 1965 make depend 1966 1967 ### Differentiate code to generate adjoint code using Tapenade 1968 ### Creates executable mitgcmuv_tap_adj 1969 make -j 8 tap_adj 1970 1971 ### Rest of the setup is standard 1972 ### These commands are for a typical verification experiment 1973 cd ../run 1974 rm -r * 1975 ln -s ../input_tap/* . 1976 ../input_tap/prepare_run 1977 ln -s ../build/mitgcmuv_tap_adj . 1978 ./mitgcmuv_tap_adj > output_tap_adj.txt 2>&1 1979 9d0c386f0c dngo*1980 .. rubric:: Footnotes 1981 1982 .. [#thanks-Dan] A big thank you to Dan Goldberg for supplying the definition 1983 file for the Singularity container!
| [ Source navigation ] | [ Diff markup ] | [ Identifier search ] | [ general search ] |
|
This page was automatically generated from https://github.com/darwinproject/darwin3 by the 2.3.7-MITgcm-0.1 LXR engine. The LXR team |
|