How to package pyi file for C extension

[kadler]

PyODBC is packaged only as a C extension and contains no Python modules/packages. In mkleehammer/pyodbc#864, typing support was added with a pyi file, but the way it was installed causes it to go to the wrong place: mkleehammer/pyodbc#925

From what I can tell, the best way to include the pyi file would be through package data, but I couldn't figure out how to do this for a C extension; it seems package_data wanted a Python module/package.

What's the recommended way to include pyi and other package data for a C extension only package?

[abravalheri]

Hi @kadler, I think I saw something like the following before:

diff --git i/setup.py w/setup.py
index 260ce7d..3988281 100755
--- i/setup.py
+++ w/setup.py
@@ -89,10 +89,8 @@ def main():
         'maintainer_email': "[email protected]",

         'ext_modules': [Extension('pyodbc', sorted(files), **settings)],
-
-        'data_files': [
-            ('', ['src/pyodbc.pyi'])  # places pyodbc.pyi alongside pyodbc.py in site-packages
-        ],
+        'packages': ["pyodbc-stubs"],
+        'package_data': {"": ["*.pyi"]},  # could also be include_package_data=True

         'license': 'MIT'

mkdir pyodbc-stubs
mv pyodbc.pyi pyodbc-stubs/__init__.pyi

Which should give you the equivalent to:

Archive:  dist/pyodbc-4.0.35b2-cp38-cp38-linux_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
   921712  2022-08-25 16:44   pyodbc.cpython-38-x86_64-linux-gnu.so
        0  2022-08-25 16:24   pyodbc-stubs/__init__.pyi
      868  2022-08-25 16:44   pyodbc-4.0.35b2.dist-info/LICENSE.txt
     2701  2022-08-25 16:44   pyodbc-4.0.35b2.dist-info/METADATA
      103  2022-08-25 16:44   pyodbc-4.0.35b2.dist-info/WHEEL
       20  2022-08-25 16:44   pyodbc-4.0.35b2.dist-info/top_level.txt
      575  2022-08-25 16:44   pyodbc-4.0.35b2.dist-info/RECORD
---------                     -------
   925979                     7 files

Would that work for you? It seems to be aligned with PEP 561...
(There might be other possibilities of implementing this also).

[kadler]

That seems like a good solution. I guess the only concern for me from reading the PEP is that the stubs package seems to be more for shipping as a separate package from the main package, which wouldn't be the case here though it doesn't sound verboten.

Reading the PEP, this did catch my eye, though:

This PEP does not support distributing typing information as part of module-only distributions or single-file modules within namespace packages.

The single-file module should be refactored into a package and indicate that the package supports typing as described above.

Sounds like this should probably also include C extension only packages as well, since it seems to have the same problem.

The resolution sounds like it would be the same as I mentioned in my latest update to mkleehammer/pyodbc#925 - create a python shim package around the C extension and package the pyi file with the python shim (though it now sounds like we'd need a package instead of just a module).

[abravalheri]

I suppose there are many ways of solving this. And they are probably going to sound very similar to your proposal in the pyodbc repository, apart from the specific details.

For example, what do you think about?

diff --git i/setup.py w/setup.py
index 260ce7d..e503530 100755
--- i/setup.py
+++ w/setup.py
@@ -88,11 +88,9 @@ def main():
         'maintainer':       "Michael Kleehammer",
         'maintainer_email': "[email protected]",

-        'ext_modules': [Extension('pyodbc', sorted(files), **settings)],
-
-        'data_files': [
-            ('', ['src/pyodbc.pyi'])  # places pyodbc.pyi alongside pyodbc.py in site-packages
-        ],
+        'ext_modules': [Extension('pyodbc.__init__', sorted(files), **settings)],
+        'packages': ["pyodbc"],
+        'package_data': {"": ["*.pyi"]},

         'license': 'MIT',

+

mkdir pyodbc
mv pyodbc.pyi pyodbc/__init__.pyi

Alternatively, you could even try something more complex:

diff --git i/setup.py w/setup.py
index 260ce7d..203a851 100755
--- i/setup.py
+++ w/setup.py
@@ -6,8 +6,10 @@ from os.path import exists, abspath, dirname, join, isdir, relpath, expanduser
 try:
     # Allow use of setuptools so eggs can be built.
     from setuptools import setup, Command
+    from setuptools.command.build import build as orig_build
 except ImportError:
     from distutils.core import setup, Command
+    from distutils.command.build import build as orig_build

 from distutils.extension import Extension
 from distutils.errors import *
@@ -62,6 +64,33 @@ class TagsCommand(Command):
         return os.system(cmd)


+class Build(orig_build):
+    # Add custom build step
+    sub_commands = [*orig_build.sub_commands, ("build_custom_files", None)]
+
+
+class BuildCustomFiles(Command):
+    editable_mode = False
+    def initialize_options(self): pass
+    def finalize_options(self): pass
+
+    def run(self):
+        if not self.editable_mode:
+            for target, src in self.get_output_mapping().items():
+                self.copy_file(src, target)
+
+    def get_source_files(self, *args, **kwargs):
+        return list(self.get_output_mapping().values())
+
+    def get_outputs(self, *args, **kwargs):
+        return list(self.get_output_mapping().keys())
+
+    def get_output_mapping(self):
+        build_lib = self.get_finalized_command("build_ext").build_lib
+        stub = os.path.join(self.distribution.src_root or "", "src", "pyodbc.pyi")
+        target = os.path.join(build_lib, "pyodbc.pyi")
+        return {target: stub}
+

 def main():

@@ -90,10 +119,6 @@ def main():

         'ext_modules': [Extension('pyodbc', sorted(files), **settings)],

-        'data_files': [
-            ('', ['src/pyodbc.pyi'])  # places pyodbc.pyi alongside pyodbc.py in site-packages
-        ],
-
         'license': 'MIT',

         'python_requires': '>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*',
@@ -117,8 +142,11 @@ def main():
                        ],

         'url': 'https://github.com/mkleehammer/pyodbc',
-        'cmdclass': { 'version' : VersionCommand,
-                     'tags'    : TagsCommand }
+        'cmdclass': {'version' : VersionCommand,
+                     'tags'    : TagsCommand,
+                     'build'   : Build,
+                     'build_custom_files': BuildCustomFiles,
+                     }
         }

     if sys.hexversion >= 0x02060000:

Sounds like this should probably also include C extension only packages as well, since it seems to have the same problem.

It might be a good idea to ask for PEP clarifications on https://discuss.python.org/c/packaging/.