The flux of [tex]\vec f(x,y,z)[/tex] across S is given by the surface integral
[tex]\displaystyle \iint_S \vec f(x,y,z) \cdot d\vec s[/tex]
Join to S the disk D of radius 3 in the plane z = 2, for which we have
[tex]x^2+y^2+2=11 \implies x^2+y^2=9=3^2[/tex]
Let S' = S U D (the union of S and D). Since S' is closed, we can use divergence theorem to compute the flux of [tex]\vec f[/tex] through S' :
[tex]\displaystyle \iint_{S'} \vec f \cdot \vec s = \iiint_R \mathrm{div}\vec f \, dV[/tex]
Compute the divergence of [tex]\vec f[/tex] :
[tex]\mathrm{div}\vec f = \dfrac{\partial\left(z\tan^{-1}(y^2)\right)}{\partial x} + \dfrac{\partial\left(z^3\ln(x^2+5)\right)}{\partial y} + \dfrac{\partial(z)}{\partial z} = 1[/tex]
Compute the volume integral by converting to cylindrical coordinates. Take
[tex]\begin{cases}x=r\cos(\theta) \\ y = r\sin(\theta) \\ z = \zeta \\ dV = dx\,dy\,dz = r\,dr\,d\theta\,d\zeta\end{cases}[/tex]
Then the flux of [tex]\vec f[/tex] across S' is
[tex]\displaystyle \iint_{S'} \vec f \cdot \vec s = \int_{-3}^3 \int_{-\sqrt{9-x^2}}^{\sqrt{9-x^2}} \int_2^{11-x^2-y^2} dV = \int_0^{2\pi} \int_0^3 \int_2^{11-r^2} r \, d\zeta \, dr \, d\theta = \frac{81\pi}2[/tex]
To get the flux across S alone, we subtract from this integral the flux of [tex]\vec f[/tex] across D.
Parameterize D by the vector function
[tex]\vec\sigma(\rho,\phi) = \rho \cos(\phi) \, \vec\imath + \rho \sin(\phi) \, \vec\jmath + 2\, \vec k[/tex]
with [tex]0\le\rho\le3[/tex] and [tex]0\le\phi\le2\pi[/tex].
Get the downward-pointing normal vector to D :
[tex]\vec n = \dfrac{\partial\vec\sigma}{\partial\phi} \times \dfrac{\partial\vec\sigma}{\partial \rho} = -\rho\,\vec k[/tex]
Compute the flux across D :
[tex]\displaystyle \iint_D \vec f\cdot d\vec s = \int_0^{2\pi} \int_0^3 \vec f(\vec\sigma) \cdot \vec n \, d\rho\,d\phi = \int_0^{2\pi} \int_0^3 (-2\rho) \, d\rho \, d\phi = -18\pi[/tex]
So the flux of [tex]\vec f[/tex] across S is
[tex]\displaystyle \iint_S \vec f \cdot d \vec s = \frac{81\pi}2 - (-18\pi) = \boxed{\frac{117\pi}2}[/tex]
