switching to high quality piper tts and added label translations

This commit is contained in:
Matthias Hinrichs
2026-01-29 23:48:19 +01:00
commit d80c619df9
3934 changed files with 1451600 additions and 0 deletions
+675
View File
@@ -0,0 +1,675 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
<program> Copyright (C) <year> <name of author>
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<http://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<http://www.gnu.org/philosophy/why-not-lgpl.html>.
+247
View File
@@ -0,0 +1,247 @@
<#
.Synopsis
Activate a Python virtual environment for the current PowerShell session.
.Description
Pushes the python executable for a virtual environment to the front of the
$Env:PATH environment variable and sets the prompt to signify that you are
in a Python virtual environment. Makes use of the command line switches as
well as the `pyvenv.cfg` file values present in the virtual environment.
.Parameter VenvDir
Path to the directory that contains the virtual environment to activate. The
default value for this is the parent of the directory that the Activate.ps1
script is located within.
.Parameter Prompt
The prompt prefix to display when this virtual environment is activated. By
default, this prompt is the name of the virtual environment folder (VenvDir)
surrounded by parentheses and followed by a single space (ie. '(.venv) ').
.Example
Activate.ps1
Activates the Python virtual environment that contains the Activate.ps1 script.
.Example
Activate.ps1 -Verbose
Activates the Python virtual environment that contains the Activate.ps1 script,
and shows extra information about the activation as it executes.
.Example
Activate.ps1 -VenvDir C:\Users\MyUser\Common\.venv
Activates the Python virtual environment located in the specified location.
.Example
Activate.ps1 -Prompt "MyPython"
Activates the Python virtual environment that contains the Activate.ps1 script,
and prefixes the current prompt with the specified string (surrounded in
parentheses) while the virtual environment is active.
.Notes
On Windows, it may be required to enable this Activate.ps1 script by setting the
execution policy for the user. You can do this by issuing the following PowerShell
command:
PS C:\> Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
For more information on Execution Policies:
https://go.microsoft.com/fwlink/?LinkID=135170
#>
Param(
[Parameter(Mandatory = $false)]
[String]
$VenvDir,
[Parameter(Mandatory = $false)]
[String]
$Prompt
)
<# Function declarations --------------------------------------------------- #>
<#
.Synopsis
Remove all shell session elements added by the Activate script, including the
addition of the virtual environment's Python executable from the beginning of
the PATH variable.
.Parameter NonDestructive
If present, do not remove this function from the global namespace for the
session.
#>
function global:deactivate ([switch]$NonDestructive) {
# Revert to original values
# The prior prompt:
if (Test-Path -Path Function:_OLD_VIRTUAL_PROMPT) {
Copy-Item -Path Function:_OLD_VIRTUAL_PROMPT -Destination Function:prompt
Remove-Item -Path Function:_OLD_VIRTUAL_PROMPT
}
# The prior PYTHONHOME:
if (Test-Path -Path Env:_OLD_VIRTUAL_PYTHONHOME) {
Copy-Item -Path Env:_OLD_VIRTUAL_PYTHONHOME -Destination Env:PYTHONHOME
Remove-Item -Path Env:_OLD_VIRTUAL_PYTHONHOME
}
# The prior PATH:
if (Test-Path -Path Env:_OLD_VIRTUAL_PATH) {
Copy-Item -Path Env:_OLD_VIRTUAL_PATH -Destination Env:PATH
Remove-Item -Path Env:_OLD_VIRTUAL_PATH
}
# Just remove the VIRTUAL_ENV altogether:
if (Test-Path -Path Env:VIRTUAL_ENV) {
Remove-Item -Path env:VIRTUAL_ENV
}
# Just remove VIRTUAL_ENV_PROMPT altogether.
if (Test-Path -Path Env:VIRTUAL_ENV_PROMPT) {
Remove-Item -Path env:VIRTUAL_ENV_PROMPT
}
# Just remove the _PYTHON_VENV_PROMPT_PREFIX altogether:
if (Get-Variable -Name "_PYTHON_VENV_PROMPT_PREFIX" -ErrorAction SilentlyContinue) {
Remove-Variable -Name _PYTHON_VENV_PROMPT_PREFIX -Scope Global -Force
}
# Leave deactivate function in the global namespace if requested:
if (-not $NonDestructive) {
Remove-Item -Path function:deactivate
}
}
<#
.Description
Get-PyVenvConfig parses the values from the pyvenv.cfg file located in the
given folder, and returns them in a map.
For each line in the pyvenv.cfg file, if that line can be parsed into exactly
two strings separated by `=` (with any amount of whitespace surrounding the =)
then it is considered a `key = value` line. The left hand string is the key,
the right hand is the value.
If the value starts with a `'` or a `"` then the first and last character is
stripped from the value before being captured.
.Parameter ConfigDir
Path to the directory that contains the `pyvenv.cfg` file.
#>
function Get-PyVenvConfig(
[String]
$ConfigDir
) {
Write-Verbose "Given ConfigDir=$ConfigDir, obtain values in pyvenv.cfg"
# Ensure the file exists, and issue a warning if it doesn't (but still allow the function to continue).
$pyvenvConfigPath = Join-Path -Resolve -Path $ConfigDir -ChildPath 'pyvenv.cfg' -ErrorAction Continue
# An empty map will be returned if no config file is found.
$pyvenvConfig = @{ }
if ($pyvenvConfigPath) {
Write-Verbose "File exists, parse `key = value` lines"
$pyvenvConfigContent = Get-Content -Path $pyvenvConfigPath
$pyvenvConfigContent | ForEach-Object {
$keyval = $PSItem -split "\s*=\s*", 2
if ($keyval[0] -and $keyval[1]) {
$val = $keyval[1]
# Remove extraneous quotations around a string value.
if ("'""".Contains($val.Substring(0, 1))) {
$val = $val.Substring(1, $val.Length - 2)
}
$pyvenvConfig[$keyval[0]] = $val
Write-Verbose "Adding Key: '$($keyval[0])'='$val'"
}
}
}
return $pyvenvConfig
}
<# Begin Activate script --------------------------------------------------- #>
# Determine the containing directory of this script
$VenvExecPath = Split-Path -Parent $MyInvocation.MyCommand.Definition
$VenvExecDir = Get-Item -Path $VenvExecPath
Write-Verbose "Activation script is located in path: '$VenvExecPath'"
Write-Verbose "VenvExecDir Fullname: '$($VenvExecDir.FullName)"
Write-Verbose "VenvExecDir Name: '$($VenvExecDir.Name)"
# Set values required in priority: CmdLine, ConfigFile, Default
# First, get the location of the virtual environment, it might not be
# VenvExecDir if specified on the command line.
if ($VenvDir) {
Write-Verbose "VenvDir given as parameter, using '$VenvDir' to determine values"
}
else {
Write-Verbose "VenvDir not given as a parameter, using parent directory name as VenvDir."
$VenvDir = $VenvExecDir.Parent.FullName.TrimEnd("\\/")
Write-Verbose "VenvDir=$VenvDir"
}
# Next, read the `pyvenv.cfg` file to determine any required value such
# as `prompt`.
$pyvenvCfg = Get-PyVenvConfig -ConfigDir $VenvDir
# Next, set the prompt from the command line, or the config file, or
# just use the name of the virtual environment folder.
if ($Prompt) {
Write-Verbose "Prompt specified as argument, using '$Prompt'"
}
else {
Write-Verbose "Prompt not specified as argument to script, checking pyvenv.cfg value"
if ($pyvenvCfg -and $pyvenvCfg['prompt']) {
Write-Verbose " Setting based on value in pyvenv.cfg='$($pyvenvCfg['prompt'])'"
$Prompt = $pyvenvCfg['prompt'];
}
else {
Write-Verbose " Setting prompt based on parent's directory's name. (Is the directory name passed to venv module when creating the virtual environment)"
Write-Verbose " Got leaf-name of $VenvDir='$(Split-Path -Path $venvDir -Leaf)'"
$Prompt = Split-Path -Path $venvDir -Leaf
}
}
Write-Verbose "Prompt = '$Prompt'"
Write-Verbose "VenvDir='$VenvDir'"
# Deactivate any currently active virtual environment, but leave the
# deactivate function in place.
deactivate -nondestructive
# Now set the environment variable VIRTUAL_ENV, used by many tools to determine
# that there is an activated venv.
$env:VIRTUAL_ENV = $VenvDir
if (-not $Env:VIRTUAL_ENV_DISABLE_PROMPT) {
Write-Verbose "Setting prompt to '$Prompt'"
# Set the prompt to include the env name
# Make sure _OLD_VIRTUAL_PROMPT is global
function global:_OLD_VIRTUAL_PROMPT { "" }
Copy-Item -Path function:prompt -Destination function:_OLD_VIRTUAL_PROMPT
New-Variable -Name _PYTHON_VENV_PROMPT_PREFIX -Description "Python virtual environment prompt prefix" -Scope Global -Option ReadOnly -Visibility Public -Value $Prompt
function global:prompt {
Write-Host -NoNewline -ForegroundColor Green "($_PYTHON_VENV_PROMPT_PREFIX) "
_OLD_VIRTUAL_PROMPT
}
$env:VIRTUAL_ENV_PROMPT = $Prompt
}
# Clear PYTHONHOME
if (Test-Path -Path Env:PYTHONHOME) {
Copy-Item -Path Env:PYTHONHOME -Destination Env:_OLD_VIRTUAL_PYTHONHOME
Remove-Item -Path Env:PYTHONHOME
}
# Add the venv to the PATH
Copy-Item -Path Env:PATH -Destination Env:_OLD_VIRTUAL_PATH
$Env:PATH = "$VenvExecDir$([System.IO.Path]::PathSeparator)$Env:PATH"
+76
View File
@@ -0,0 +1,76 @@
# This file must be used with "source bin/activate" *from bash*
# You cannot run it directly
deactivate () {
# reset old environment variables
if [ -n "${_OLD_VIRTUAL_PATH:-}" ] ; then
PATH="${_OLD_VIRTUAL_PATH:-}"
export PATH
unset _OLD_VIRTUAL_PATH
fi
if [ -n "${_OLD_VIRTUAL_PYTHONHOME:-}" ] ; then
PYTHONHOME="${_OLD_VIRTUAL_PYTHONHOME:-}"
export PYTHONHOME
unset _OLD_VIRTUAL_PYTHONHOME
fi
# Call hash to forget past locations. Without forgetting
# past locations the $PATH changes we made may not be respected.
# See "man bash" for more details. hash is usually a builtin of your shell
hash -r 2> /dev/null
if [ -n "${_OLD_VIRTUAL_PS1:-}" ] ; then
PS1="${_OLD_VIRTUAL_PS1:-}"
export PS1
unset _OLD_VIRTUAL_PS1
fi
unset VIRTUAL_ENV
unset VIRTUAL_ENV_PROMPT
if [ ! "${1:-}" = "nondestructive" ] ; then
# Self destruct!
unset -f deactivate
fi
}
# unset irrelevant variables
deactivate nondestructive
# on Windows, a path can contain colons and backslashes and has to be converted:
case "$(uname)" in
CYGWIN*|MSYS*|MINGW*)
# transform D:\path\to\venv to /d/path/to/venv on MSYS and MINGW
# and to /cygdrive/d/path/to/venv on Cygwin
VIRTUAL_ENV=$(cygpath /Users/matthias/Documents/Projects/Homelab/Home-Bro/home-bro-brain/venv_piper)
export VIRTUAL_ENV
;;
*)
# use the path as-is
export VIRTUAL_ENV=/Users/matthias/Documents/Projects/Homelab/Home-Bro/home-bro-brain/venv_piper
;;
esac
_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/"bin":$PATH"
export PATH
VIRTUAL_ENV_PROMPT='(venv_piper) '
export VIRTUAL_ENV_PROMPT
# unset PYTHONHOME if set
# this will fail if PYTHONHOME is set to the empty string (which is bad anyway)
# could use `if (set -u; : $PYTHONHOME) ;` in bash
if [ -n "${PYTHONHOME:-}" ] ; then
_OLD_VIRTUAL_PYTHONHOME="${PYTHONHOME:-}"
unset PYTHONHOME
fi
if [ -z "${VIRTUAL_ENV_DISABLE_PROMPT:-}" ] ; then
_OLD_VIRTUAL_PS1="${PS1:-}"
PS1="("'(venv_piper) '") ${PS1:-}"
export PS1
fi
# Call hash to forget past commands. Without forgetting
# past commands the $PATH changes we made may not be respected
hash -r 2> /dev/null
+27
View File
@@ -0,0 +1,27 @@
# This file must be used with "source bin/activate.csh" *from csh*.
# You cannot run it directly.
# Created by Davide Di Blasi <davidedb@gmail.com>.
# Ported to Python 3.3 venv by Andrew Svetlov <andrew.svetlov@gmail.com>
alias deactivate 'test $?_OLD_VIRTUAL_PATH != 0 && setenv PATH "$_OLD_VIRTUAL_PATH" && unset _OLD_VIRTUAL_PATH; rehash; test $?_OLD_VIRTUAL_PROMPT != 0 && set prompt="$_OLD_VIRTUAL_PROMPT" && unset _OLD_VIRTUAL_PROMPT; unsetenv VIRTUAL_ENV; unsetenv VIRTUAL_ENV_PROMPT; test "\!:*" != "nondestructive" && unalias deactivate'
# Unset irrelevant variables.
deactivate nondestructive
setenv VIRTUAL_ENV /Users/matthias/Documents/Projects/Homelab/Home-Bro/home-bro-brain/venv_piper
set _OLD_VIRTUAL_PATH="$PATH"
setenv PATH "$VIRTUAL_ENV/"bin":$PATH"
set _OLD_VIRTUAL_PROMPT="$prompt"
if (! "$?VIRTUAL_ENV_DISABLE_PROMPT") then
set prompt = '(venv_piper) '"$prompt"
setenv VIRTUAL_ENV_PROMPT '(venv_piper) '
endif
alias pydoc python -m pydoc
rehash
+69
View File
@@ -0,0 +1,69 @@
# This file must be used with "source <venv>/bin/activate.fish" *from fish*
# (https://fishshell.com/). You cannot run it directly.
function deactivate -d "Exit virtual environment and return to normal shell environment"
# reset old environment variables
if test -n "$_OLD_VIRTUAL_PATH"
set -gx PATH $_OLD_VIRTUAL_PATH
set -e _OLD_VIRTUAL_PATH
end
if test -n "$_OLD_VIRTUAL_PYTHONHOME"
set -gx PYTHONHOME $_OLD_VIRTUAL_PYTHONHOME
set -e _OLD_VIRTUAL_PYTHONHOME
end
if test -n "$_OLD_FISH_PROMPT_OVERRIDE"
set -e _OLD_FISH_PROMPT_OVERRIDE
# prevents error when using nested fish instances (Issue #93858)
if functions -q _old_fish_prompt
functions -e fish_prompt
functions -c _old_fish_prompt fish_prompt
functions -e _old_fish_prompt
end
end
set -e VIRTUAL_ENV
set -e VIRTUAL_ENV_PROMPT
if test "$argv[1]" != "nondestructive"
# Self-destruct!
functions -e deactivate
end
end
# Unset irrelevant variables.
deactivate nondestructive
set -gx VIRTUAL_ENV /Users/matthias/Documents/Projects/Homelab/Home-Bro/home-bro-brain/venv_piper
set -gx _OLD_VIRTUAL_PATH $PATH
set -gx PATH "$VIRTUAL_ENV/"bin $PATH
# Unset PYTHONHOME if set.
if set -q PYTHONHOME
set -gx _OLD_VIRTUAL_PYTHONHOME $PYTHONHOME
set -e PYTHONHOME
end
if test -z "$VIRTUAL_ENV_DISABLE_PROMPT"
# fish uses a function instead of an env var to generate the prompt.
# Save the current fish_prompt function as the function _old_fish_prompt.
functions -c fish_prompt _old_fish_prompt
# With the original prompt function renamed, we can override with our own.
function fish_prompt
# Save the return status of the last command.
set -l old_status $status
# Output the venv prompt; color taken from the blue of the Python logo.
printf "%s%s%s" (set_color 4B8BBE) '(venv_piper) ' (set_color normal)
# Restore the return status of the previous command.
echo "exit $old_status" | .
# Output the original/"old" prompt.
_old_fish_prompt
end
set -gx _OLD_FISH_PROMPT_OVERRIDE "$VIRTUAL_ENV"
set -gx VIRTUAL_ENV_PROMPT '(venv_piper) '
end
+7
View File
@@ -0,0 +1,7 @@
#!/Users/matthias/Documents/Projects/Homelab/Home-Bro/home-bro-brain/venv_piper/bin/python
import sys
from coloredlogs.cli import main
if __name__ == '__main__':
if sys.argv[0].endswith('.exe'):
sys.argv[0] = sys.argv[0][:-4]
sys.exit(main())
+7
View File
@@ -0,0 +1,7 @@
#!/Users/matthias/Documents/Projects/Homelab/Home-Bro/home-bro-brain/venv_piper/bin/python
import sys
from numpy.f2py.f2py2e import main
if __name__ == '__main__':
if sys.argv[0].endswith('.exe'):
sys.argv[0] = sys.argv[0][:-4]
sys.exit(main())
+7
View File
@@ -0,0 +1,7 @@
#!/Users/matthias/Documents/Projects/Homelab/Home-Bro/home-bro-brain/venv_piper/bin/python
import sys
from humanfriendly.cli import main
if __name__ == '__main__':
if sys.argv[0].endswith('.exe'):
sys.argv[0] = sys.argv[0][:-4]
sys.exit(main())
+7
View File
@@ -0,0 +1,7 @@
#!/Users/matthias/Documents/Projects/Homelab/Home-Bro/home-bro-brain/venv_piper/bin/python
import sys
from isympy import main
if __name__ == '__main__':
if sys.argv[0].endswith('.exe'):
sys.argv[0] = sys.argv[0][:-4]
sys.exit(main())
+7
View File
@@ -0,0 +1,7 @@
#!/Users/matthias/Documents/Projects/Homelab/Home-Bro/home-bro-brain/venv_piper/bin/python
import sys
from numpy._configtool import main
if __name__ == '__main__':
if sys.argv[0].endswith('.exe'):
sys.argv[0] = sys.argv[0][:-4]
sys.exit(main())
+7
View File
@@ -0,0 +1,7 @@
#!/Users/matthias/Documents/Projects/Homelab/Home-Bro/home-bro-brain/venv_piper/bin/python
import sys
from onnxruntime.tools.onnxruntime_test import main
if __name__ == '__main__':
if sys.argv[0].endswith('.exe'):
sys.argv[0] = sys.argv[0][:-4]
sys.exit(main())
+7
View File
@@ -0,0 +1,7 @@
#!/Users/matthias/Documents/Projects/Homelab/Home-Bro/home-bro-brain/venv_piper/bin/python3.12
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
if sys.argv[0].endswith('.exe'):
sys.argv[0] = sys.argv[0][:-4]
sys.exit(main())
+7
View File
@@ -0,0 +1,7 @@
#!/Users/matthias/Documents/Projects/Homelab/Home-Bro/home-bro-brain/venv_piper/bin/python3.12
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
if sys.argv[0].endswith('.exe'):
sys.argv[0] = sys.argv[0][:-4]
sys.exit(main())
+7
View File
@@ -0,0 +1,7 @@
#!/Users/matthias/Documents/Projects/Homelab/Home-Bro/home-bro-brain/venv_piper/bin/python3.12
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
if sys.argv[0].endswith('.exe'):
sys.argv[0] = sys.argv[0][:-4]
sys.exit(main())
+7
View File
@@ -0,0 +1,7 @@
#!/Users/matthias/Documents/Projects/Homelab/Home-Bro/home-bro-brain/venv_piper/bin/python
import sys
from piper.__main__ import main
if __name__ == '__main__':
if sys.argv[0].endswith('.exe'):
sys.argv[0] = sys.argv[0][:-4]
sys.exit(main())
+1
View File
@@ -0,0 +1 @@
python3.12
+1
View File
@@ -0,0 +1 @@
python3.12
+1
View File
@@ -0,0 +1 @@
/opt/homebrew/opt/python@3.12/bin/python3.12
@@ -0,0 +1,20 @@
Copyright (c) 2020 Peter Odding
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,291 @@
Metadata-Version: 2.1
Name: coloredlogs
Version: 15.0.1
Summary: Colored terminal output for Python's logging module
Home-page: https://coloredlogs.readthedocs.io
Author: Peter Odding
Author-email: peter@peterodding.com
License: MIT
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX
Classifier: Operating System :: Unix
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Communications
Classifier: Topic :: Scientific/Engineering :: Human Machine Interfaces
Classifier: Topic :: Software Development
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Software Development :: User Interfaces
Classifier: Topic :: System
Classifier: Topic :: System :: Shells
Classifier: Topic :: System :: System Shells
Classifier: Topic :: System :: Console Fonts
Classifier: Topic :: System :: Logging
Classifier: Topic :: System :: Systems Administration
Classifier: Topic :: Terminals
Classifier: Topic :: Utilities
Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*
Requires-Dist: humanfriendly (>=9.1)
Provides-Extra: cron
Requires-Dist: capturer (>=2.4) ; extra == 'cron'
coloredlogs: Colored terminal output for Python's logging module
================================================================
.. image:: https://travis-ci.org/xolox/python-coloredlogs.svg?branch=master
:target: https://travis-ci.org/xolox/python-coloredlogs
.. image:: https://coveralls.io/repos/github/xolox/python-coloredlogs/badge.svg?branch=master
:target: https://coveralls.io/github/xolox/python-coloredlogs?branch=master
The `coloredlogs` package enables colored terminal output for Python's logging_
module. The ColoredFormatter_ class inherits from `logging.Formatter`_ and uses
`ANSI escape sequences`_ to render your logging messages in color. It uses only
standard colors so it should work on any UNIX terminal. It's currently tested
on Python 2.7, 3.5+ and PyPy (2 and 3). On Windows `coloredlogs` automatically
tries to enable native ANSI support (on up-to-date Windows 10 installations)
and falls back on using colorama_ (if installed). Here is a screen shot of the
demo that is printed when the command ``coloredlogs --demo`` is executed:
.. image:: https://coloredlogs.readthedocs.io/en/latest/_images/defaults.png
Note that the screenshot above includes custom logging levels defined by my
verboselogs_ package: if you install both `coloredlogs` and `verboselogs` it
will Just Work (`verboselogs` is of course not required to use
`coloredlogs`).
.. contents::
:local:
Installation
------------
The `coloredlogs` package is available on PyPI_ which means installation should
be as simple as:
.. code-block:: console
$ pip install coloredlogs
There's actually a multitude of ways to install Python packages (e.g. the `per
user site-packages directory`_, `virtual environments`_ or just installing
system wide) and I have no intention of getting into that discussion here, so
if this intimidates you then read up on your options before returning to these
instructions 😉.
Optional dependencies
~~~~~~~~~~~~~~~~~~~~~
Native ANSI support on Windows requires an up-to-date Windows 10 installation.
If this is not working for you then consider installing the colorama_ package:
.. code-block:: console
$ pip install colorama
Once colorama_ is installed it will be used automatically.
Usage
-----
Here's an example of how easy it is to get started:
.. code-block:: python
import coloredlogs, logging
# Create a logger object.
logger = logging.getLogger(__name__)
# By default the install() function installs a handler on the root logger,
# this means that log messages from your code and log messages from the
# libraries that you use will all show up on the terminal.
coloredlogs.install(level='DEBUG')
# If you don't want to see log messages from libraries, you can pass a
# specific logger object to the install() function. In this case only log
# messages originating from that logger will show up on the terminal.
coloredlogs.install(level='DEBUG', logger=logger)
# Some examples.
logger.debug("this is a debugging message")
logger.info("this is an informational message")
logger.warning("this is a warning message")
logger.error("this is an error message")
logger.critical("this is a critical message")
Format of log messages
----------------------
The ColoredFormatter_ class supports user defined log formats so you can use
any log format you like. The default log format is as follows::
%(asctime)s %(hostname)s %(name)s[%(process)d] %(levelname)s %(message)s
This log format results in the following output::
2015-10-23 03:32:22 peter-macbook coloredlogs.demo[30462] DEBUG message with level 'debug'
2015-10-23 03:32:23 peter-macbook coloredlogs.demo[30462] VERBOSE message with level 'verbose'
2015-10-23 03:32:24 peter-macbook coloredlogs.demo[30462] INFO message with level 'info'
...
You can customize the log format and styling using environment variables as
well as programmatically, please refer to the `online documentation`_ for
details.
Enabling millisecond precision
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you're switching from `logging.basicConfig()`_ to `coloredlogs.install()`_
you may notice that timestamps no longer include milliseconds. This is because
coloredlogs doesn't output milliseconds in timestamps unless you explicitly
tell it to. There are three ways to do that:
1. The easy way is to pass the `milliseconds` argument to `coloredlogs.install()`_::
coloredlogs.install(milliseconds=True)
This became supported in `release 7.1`_ (due to `#16`_).
2. Alternatively you can change the log format `to include 'msecs'`_::
%(asctime)s,%(msecs)03d %(hostname)s %(name)s[%(process)d] %(levelname)s %(message)s
Here's what the call to `coloredlogs.install()`_ would then look like::
coloredlogs.install(fmt='%(asctime)s,%(msecs)03d %(hostname)s %(name)s[%(process)d] %(levelname)s %(message)s')
Customizing the log format also enables you to change the delimiter that
separates seconds from milliseconds (the comma above). This became possible
in `release 3.0`_ which added support for user defined log formats.
3. If the use of ``%(msecs)d`` isn't flexible enough you can instead add ``%f``
to the date/time format, it will be replaced by the value of ``%(msecs)03d``.
Support for the ``%f`` directive was added to `release 9.3`_ (due to `#45`_).
Custom logging fields
~~~~~~~~~~~~~~~~~~~~~
The following custom log format fields are supported:
- ``%(hostname)s`` provides the hostname of the local system.
- ``%(programname)s`` provides the name of the currently running program.
- ``%(username)s`` provides the username of the currently logged in user.
When `coloredlogs.install()`_ detects that any of these fields are used in the
format string the applicable logging.Filter_ subclasses are automatically
registered to populate the relevant log record fields.
Changing text styles and colors
-------------------------------
The online documentation contains `an example of customizing the text styles and
colors <https://coloredlogs.readthedocs.io/en/latest/api.html#changing-the-colors-styles>`_.
Colored output from cron
------------------------
When `coloredlogs` is used in a cron_ job, the output that's e-mailed to you by
cron won't contain any ANSI escape sequences because `coloredlogs` realizes
that it's not attached to an interactive terminal. If you'd like to have colors
e-mailed to you by cron there are two ways to make it happen:
.. contents::
:local:
Modifying your crontab
~~~~~~~~~~~~~~~~~~~~~~
Here's an example of a minimal crontab::
MAILTO="your-email-address@here"
CONTENT_TYPE="text/html"
* * * * * root coloredlogs --to-html your-command
The ``coloredlogs`` program is installed when you install the `coloredlogs`
Python package. When you execute ``coloredlogs --to-html your-command`` it runs
``your-command`` under the external program ``script`` (you need to have this
installed). This makes ``your-command`` think that it's attached to an
interactive terminal which means it will output ANSI escape sequences which
will then be converted to HTML by the ``coloredlogs`` program. Yes, this is a
bit convoluted, but it works great :-)
Modifying your Python code
~~~~~~~~~~~~~~~~~~~~~~~~~~
The ColoredCronMailer_ class provides a context manager that automatically
enables HTML output when the ``$CONTENT_TYPE`` variable has been correctly set
in the crontab.
This requires my capturer_ package which you can install using ``pip install
'coloredlogs[cron]'``. The ``[cron]`` extra will pull in capturer_ 2.4 or newer
which is required to capture the output while silencing it - otherwise you'd
get duplicate output in the emails sent by ``cron``.
The context manager can also be used to retroactively silence output that has
already been produced, this can be useful to avoid spammy cron jobs that have
nothing useful to do but still email their output to the system administrator
every few minutes :-).
Contact
-------
The latest version of `coloredlogs` is available on PyPI_ and GitHub_. The
`online documentation`_ is available on Read The Docs and includes a
changelog_. For bug reports please create an issue on GitHub_. If you have
questions, suggestions, etc. feel free to send me an e-mail at
`peter@peterodding.com`_.
License
-------
This software is licensed under the `MIT license`_.
© 2020 Peter Odding.
.. External references:
.. _#16: https://github.com/xolox/python-coloredlogs/issues/16
.. _#45: https://github.com/xolox/python-coloredlogs/issues/45
.. _ANSI escape sequences: https://en.wikipedia.org/wiki/ANSI_escape_code#Colors
.. _capturer: https://pypi.org/project/capturer
.. _changelog: https://coloredlogs.readthedocs.org/en/latest/changelog.html
.. _colorama: https://pypi.org/project/colorama
.. _ColoredCronMailer: https://coloredlogs.readthedocs.io/en/latest/api.html#coloredlogs.converter.ColoredCronMailer
.. _ColoredFormatter: https://coloredlogs.readthedocs.io/en/latest/api.html#coloredlogs.ColoredFormatter
.. _coloredlogs.install(): https://coloredlogs.readthedocs.io/en/latest/api.html#coloredlogs.install
.. _cron: https://en.wikipedia.org/wiki/Cron
.. _GitHub: https://github.com/xolox/python-coloredlogs
.. _logging.basicConfig(): https://docs.python.org/2/library/logging.html#logging.basicConfig
.. _logging.Filter: https://docs.python.org/3/library/logging.html#filter-objects
.. _logging.Formatter: https://docs.python.org/2/library/logging.html#logging.Formatter
.. _logging: https://docs.python.org/2/library/logging.html
.. _MIT license: https://en.wikipedia.org/wiki/MIT_License
.. _online documentation: https://coloredlogs.readthedocs.io/
.. _per user site-packages directory: https://www.python.org/dev/peps/pep-0370/
.. _peter@peterodding.com: peter@peterodding.com
.. _PyPI: https://pypi.org/project/coloredlogs
.. _release 3.0: https://coloredlogs.readthedocs.io/en/latest/changelog.html#release-3-0-2015-10-23
.. _release 7.1: https://coloredlogs.readthedocs.io/en/latest/changelog.html#release-7-1-2017-07-15
.. _release 9.3: https://coloredlogs.readthedocs.io/en/latest/changelog.html#release-9-3-2018-04-29
.. _to include 'msecs': https://stackoverflow.com/questions/6290739/python-logging-use-milliseconds-in-time-format
.. _verboselogs: https://pypi.org/project/verboselogs
.. _virtual environments: http://docs.python-guide.org/en/latest/dev/virtualenvs/
@@ -0,0 +1,23 @@
../../../bin/coloredlogs,sha256=q-s7Du259fyzdKUNCF3nvGID0d2Ikl-6lvZd83I8vLY,259
coloredlogs-15.0.1.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
coloredlogs-15.0.1.dist-info/LICENSE.txt,sha256=C-9P-xhVtlVwME8gGMbUTkurl71d-HFuzXgAAU1xcmc,1056
coloredlogs-15.0.1.dist-info/METADATA,sha256=FmO6unRvNe77JJ2UU0XYhWbMTzZDhj6zGpEljDysZ0w,12387
coloredlogs-15.0.1.dist-info/RECORD,,
coloredlogs-15.0.1.dist-info/WHEEL,sha256=kGT74LWyRUZrL4VgLh6_g12IeVl_9u9ZVhadrgXZUEY,110
coloredlogs-15.0.1.dist-info/entry_points.txt,sha256=uOmYMPjy7X6effmVh56i3U3N9S13DTRozJOAncBtZG0,54
coloredlogs-15.0.1.dist-info/top_level.txt,sha256=D3LuRvusEQ8xKUhPowPUmPWNm88FSw8ts3x2ulvCSyQ,12
coloredlogs.pth,sha256=3ag6hVmG76XNh_AkiwGZwAhusOjn_s59Z0GVnFzjlTY,147
coloredlogs/__init__.py,sha256=FxaiI1pQ0JzRYbx2K5f_Ar9Wa1meZLg5XKGWHKgPBNo,64423
coloredlogs/__pycache__/__init__.cpython-312.pyc,,
coloredlogs/__pycache__/cli.cpython-312.pyc,,
coloredlogs/__pycache__/demo.cpython-312.pyc,,
coloredlogs/__pycache__/syslog.cpython-312.pyc,,
coloredlogs/__pycache__/tests.cpython-312.pyc,,
coloredlogs/cli.py,sha256=iaHgUVeHPl5iIecKG1WJMslOJEKdhEusbxj7B4r1sHI,3493
coloredlogs/converter/__init__.py,sha256=t08e9V8-Ed9y9eNSYv3x1DH0hXgaXHJmQWp54j_a2nk,18353
coloredlogs/converter/__pycache__/__init__.cpython-312.pyc,,
coloredlogs/converter/__pycache__/colors.cpython-312.pyc,,
coloredlogs/converter/colors.py,sha256=1N2PpCa-EYMMWyC6Dw7SG2WX1gC67F9F5OteeFW52SM,5387
coloredlogs/demo.py,sha256=CaPVGOB6rCtJDyeqk0VBLi0t05jgUEniuB__VYo5kho,1825
coloredlogs/syslog.py,sha256=-XyRUzI20QKTTtVqhJAtfYc-ljw0crsjxqjWDjqrcqU,11849
coloredlogs/tests.py,sha256=oe_pWnTGfNEI7spY3_VA62fX_09ZEVFVERvr9sBMqnM,30791
@@ -0,0 +1,6 @@
Wheel-Version: 1.0
Generator: bdist_wheel (0.34.2)
Root-Is-Purelib: true
Tag: py2-none-any
Tag: py3-none-any
@@ -0,0 +1,3 @@
[console_scripts]
coloredlogs = coloredlogs.cli:main
@@ -0,0 +1 @@
import os; exec('try: __import__("coloredlogs").auto_install() if os.environ.get("COLOREDLOGS_AUTO_INSTALL") else None\nexcept ImportError: pass')
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,106 @@
# Command line interface for the coloredlogs package.
#
# Author: Peter Odding <peter@peterodding.com>
# Last Change: December 15, 2017
# URL: https://coloredlogs.readthedocs.io
"""
Usage: coloredlogs [OPTIONS] [ARGS]
The coloredlogs program provides a simple command line interface for the Python
package by the same name.
Supported options:
-c, --convert, --to-html
Capture the output of an external command (given by the positional
arguments) and convert ANSI escape sequences in the output to HTML.
If the `coloredlogs' program is attached to an interactive terminal it will
write the generated HTML to a temporary file and open that file in a web
browser, otherwise the generated HTML will be written to standard output.
This requires the `script' program to fake the external command into
thinking that it's attached to an interactive terminal (in order to enable
output of ANSI escape sequences).
If the command didn't produce any output then no HTML will be produced on
standard output, this is to avoid empty emails from cron jobs.
-d, --demo
Perform a simple demonstration of the coloredlogs package to show the
colored logging on an interactive terminal.
-h, --help
Show this message and exit.
"""
# Standard library modules.
import functools
import getopt
import logging
import sys
import tempfile
import webbrowser
# External dependencies.
from humanfriendly.terminal import connected_to_terminal, output, usage, warning
# Modules included in our package.
from coloredlogs.converter import capture, convert
from coloredlogs.demo import demonstrate_colored_logging
# Initialize a logger for this module.
logger = logging.getLogger(__name__)
def main():
"""Command line interface for the ``coloredlogs`` program."""
actions = []
try:
# Parse the command line arguments.
options, arguments = getopt.getopt(sys.argv[1:], 'cdh', [
'convert', 'to-html', 'demo', 'help',
])
# Map command line options to actions.
for option, value in options:
if option in ('-c', '--convert', '--to-html'):
actions.append(functools.partial(convert_command_output, *arguments))
arguments = []
elif option in ('-d', '--demo'):
actions.append(demonstrate_colored_logging)
elif option in ('-h', '--help'):
usage(__doc__)
return
else:
assert False, "Programming error: Unhandled option!"
if not actions:
usage(__doc__)
return
except Exception as e:
warning("Error: %s", e)
sys.exit(1)
for function in actions:
function()
def convert_command_output(*command):
"""
Command line interface for ``coloredlogs --to-html``.
Takes a command (and its arguments) and runs the program under ``script``
(emulating an interactive terminal), intercepts the output of the command
and converts ANSI escape sequences in the output to HTML.
"""
captured_output = capture(command)
converted_output = convert(captured_output)
if connected_to_terminal():
fd, temporary_file = tempfile.mkstemp(suffix='.html')
with open(temporary_file, 'w') as handle:
handle.write(converted_output)
webbrowser.open(temporary_file)
elif captured_output and not captured_output.isspace():
output(converted_output)
@@ -0,0 +1,403 @@
# Program to convert text with ANSI escape sequences to HTML.
#
# Author: Peter Odding <peter@peterodding.com>
# Last Change: February 14, 2020
# URL: https://coloredlogs.readthedocs.io
"""Convert text with ANSI escape sequences to HTML."""
# Standard library modules.
import codecs
import os
import pipes
import re
import subprocess
import tempfile
# External dependencies.
from humanfriendly.terminal import (
ANSI_CSI,
ANSI_TEXT_STYLES,
clean_terminal_output,
output,
)
# Modules included in our package.
from coloredlogs.converter.colors import (
BRIGHT_COLOR_PALETTE,
EIGHT_COLOR_PALETTE,
EXTENDED_COLOR_PALETTE,
)
# Compiled regular expression that matches leading spaces (indentation).
INDENT_PATTERN = re.compile('^ +', re.MULTILINE)
# Compiled regular expression that matches a tag followed by a space at the start of a line.
TAG_INDENT_PATTERN = re.compile('^(<[^>]+>) ', re.MULTILINE)
# Compiled regular expression that matches strings we want to convert. Used to
# separate all special strings and literal output in a single pass (this allows
# us to properly encode the output without resorting to nasty hacks).
TOKEN_PATTERN = re.compile(r'''
# Wrap the pattern in a capture group so that re.split() includes the
# substrings that match the pattern in the resulting list of strings.
(
# Match URLs with supported schemes and domain names.
(?: https?:// | www\\. )
# Scan until the end of the URL by matching non-whitespace characters
# that are also not escape characters.
[^\s\x1b]+
# Alternatively ...
|
# Match (what looks like) ANSI escape sequences.
\x1b \[ .*? m
)
''', re.UNICODE | re.VERBOSE)
def capture(command, encoding='UTF-8'):
"""
Capture the output of an external command as if it runs in an interactive terminal.
:param command: The command name and its arguments (a list of strings).
:param encoding: The encoding to use to decode the output (a string).
:returns: The output of the command.
This function runs an external command under ``script`` (emulating an
interactive terminal) to capture the output of the command as if it was
running in an interactive terminal (including ANSI escape sequences).
"""
with open(os.devnull, 'wb') as dev_null:
# We start by invoking the `script' program in a form that is supported
# by the Linux implementation [1] but fails command line validation on
# the MacOS (BSD) implementation [2]: The command is specified using
# the -c option and the typescript file is /dev/null.
#
# [1] http://man7.org/linux/man-pages/man1/script.1.html
# [2] https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/script.1.html
command_line = ['script', '-qc', ' '.join(map(pipes.quote, command)), '/dev/null']
script = subprocess.Popen(command_line, stdout=subprocess.PIPE, stderr=dev_null)
stdout, stderr = script.communicate()
if script.returncode == 0:
# If `script' succeeded we assume that it understood our command line
# invocation which means it's the Linux implementation (in this case
# we can use standard output instead of a temporary file).
output = stdout.decode(encoding)
else:
# If `script' failed we assume that it didn't understand our command
# line invocation which means it's the MacOS (BSD) implementation
# (in this case we need a temporary file because the command line
# interface requires it).
fd, temporary_file = tempfile.mkstemp(prefix='coloredlogs-', suffix='-capture.txt')
try:
command_line = ['script', '-q', temporary_file] + list(command)
subprocess.Popen(command_line, stdout=dev_null, stderr=dev_null).wait()
with codecs.open(temporary_file, 'rb') as handle:
output = handle.read()
finally:
os.unlink(temporary_file)
# On MacOS when standard input is /dev/null I've observed
# the captured output starting with the characters '^D':
#
# $ script -q capture.txt echo example </dev/null
# example
# $ xxd capture.txt
# 00000000: 5e44 0808 6578 616d 706c 650d 0a ^D..example..
#
# I'm not sure why this is here, although I suppose it has to do
# with ^D in caret notation signifying end-of-file [1]. What I do
# know is that this is an implementation detail that callers of the
# capture() function shouldn't be bothered with, so we strip it.
#
# [1] https://en.wikipedia.org/wiki/End-of-file
if output.startswith(b'^D'):
output = output[2:]
output = output.decode(encoding)
# Clean up backspace and carriage return characters and the 'erase line'
# ANSI escape sequence and return the output as a Unicode string.
return u'\n'.join(clean_terminal_output(output))
def convert(text, code=True, tabsize=4):
"""
Convert text with ANSI escape sequences to HTML.
:param text: The text with ANSI escape sequences (a string).
:param code: Whether to wrap the returned HTML fragment in a
``<code>...</code>`` element (a boolean, defaults
to :data:`True`).
:param tabsize: Refer to :func:`str.expandtabs()` for details.
:returns: The text converted to HTML (a string).
"""
output = []
in_span = False
compatible_text_styles = {
# The following ANSI text styles have an obvious mapping to CSS.
ANSI_TEXT_STYLES['bold']: {'font-weight': 'bold'},
ANSI_TEXT_STYLES['strike_through']: {'text-decoration': 'line-through'},
ANSI_TEXT_STYLES['underline']: {'text-decoration': 'underline'},
}
for token in TOKEN_PATTERN.split(text):
if token.startswith(('http://', 'https://', 'www.')):
url = token if '://' in token else ('http://' + token)
token = u'<a href="%s" style="color:inherit">%s</a>' % (html_encode(url), html_encode(token))
elif token.startswith(ANSI_CSI):
ansi_codes = token[len(ANSI_CSI):-1].split(';')
if all(c.isdigit() for c in ansi_codes):
ansi_codes = list(map(int, ansi_codes))
# First we check for a reset code to close the previous <span>
# element. As explained on Wikipedia [1] an absence of codes
# implies a reset code as well: "No parameters at all in ESC[m acts
# like a 0 reset code".
# [1] https://en.wikipedia.org/wiki/ANSI_escape_code#CSI_sequences
if in_span and (0 in ansi_codes or not ansi_codes):
output.append('</span>')
in_span = False
# Now we're ready to generate the next <span> element (if any) in
# the knowledge that we're emitting opening <span> and closing
# </span> tags in the correct order.
styles = {}
is_faint = (ANSI_TEXT_STYLES['faint'] in ansi_codes)
is_inverse = (ANSI_TEXT_STYLES['inverse'] in ansi_codes)
while ansi_codes:
number = ansi_codes.pop(0)
# Try to match a compatible text style.
if number in compatible_text_styles:
styles.update(compatible_text_styles[number])
continue
# Try to extract a text and/or background color.
text_color = None
background_color = None
if 30 <= number <= 37:
# 30-37 sets the text color from the eight color palette.
text_color = EIGHT_COLOR_PALETTE[number - 30]
elif 40 <= number <= 47:
# 40-47 sets the background color from the eight color palette.
background_color = EIGHT_COLOR_PALETTE[number - 40]
elif 90 <= number <= 97:
# 90-97 sets the text color from the high-intensity eight color palette.
text_color = BRIGHT_COLOR_PALETTE[number - 90]
elif 100 <= number <= 107:
# 100-107 sets the background color from the high-intensity eight color palette.
background_color = BRIGHT_COLOR_PALETTE[number - 100]
elif number in (38, 39) and len(ansi_codes) >= 2 and ansi_codes[0] == 5:
# 38;5;N is a text color in the 256 color mode palette,
# 39;5;N is a background color in the 256 color mode palette.
try:
# Consume the 5 following 38 or 39.
ansi_codes.pop(0)
# Consume the 256 color mode color index.
color_index = ansi_codes.pop(0)
# Set the variable to the corresponding HTML/CSS color.
if number == 38:
text_color = EXTENDED_COLOR_PALETTE[color_index]
elif number == 39:
background_color = EXTENDED_COLOR_PALETTE[color_index]
except (ValueError, IndexError):
pass
# Apply the 'faint' or 'inverse' text style
# by manipulating the selected color(s).
if text_color and is_inverse:
# Use the text color as the background color and pick a
# text color that will be visible on the resulting
# background color.
background_color = text_color
text_color = select_text_color(*parse_hex_color(text_color))
if text_color and is_faint:
# Because I wasn't sure how to implement faint colors
# based on normal colors I looked at how gnome-terminal
# (my terminal of choice) handles this and it appears
# to just pick a somewhat darker color.
text_color = '#%02X%02X%02X' % tuple(
max(0, n - 40) for n in parse_hex_color(text_color)
)
if text_color:
styles['color'] = text_color
if background_color:
styles['background-color'] = background_color
if styles:
token = '<span style="%s">' % ';'.join(k + ':' + v for k, v in sorted(styles.items()))
in_span = True
else:
token = ''
else:
token = html_encode(token)
output.append(token)
html = ''.join(output)
html = encode_whitespace(html, tabsize)
if code:
html = '<code>%s</code>' % html
return html
def encode_whitespace(text, tabsize=4):
"""
Encode whitespace so that web browsers properly render it.
:param text: The plain text (a string).
:param tabsize: Refer to :func:`str.expandtabs()` for details.
:returns: The text converted to HTML (a string).
The purpose of this function is to encode whitespace in such a way that web
browsers render the same whitespace regardless of whether 'preformatted'
styling is used (by wrapping the text in a ``<pre>...</pre>`` element).
.. note:: While the string manipulation performed by this function is
specifically intended not to corrupt the HTML generated by
:func:`convert()` it definitely does have the potential to
corrupt HTML from other sources. You have been warned :-).
"""
# Convert Windows line endings (CR+LF) to UNIX line endings (LF).
text = text.replace('\r\n', '\n')
# Convert UNIX line endings (LF) to HTML line endings (<br>).
text = text.replace('\n', '<br>\n')
# Convert tabs to spaces.
text = text.expandtabs(tabsize)
# Convert leading spaces (that is to say spaces at the start of the string
# and/or directly after a line ending) into non-breaking spaces, otherwise
# HTML rendering engines will simply ignore these spaces.
text = re.sub(INDENT_PATTERN, encode_whitespace_cb, text)
# The conversion of leading spaces we just did misses a corner case where a
# line starts with an HTML tag but the first visible text is a space. Web
# browsers seem to ignore these spaces, so we need to convert them.
text = re.sub(TAG_INDENT_PATTERN, r'\1&nbsp;', text)
# Convert runs of multiple spaces into non-breaking spaces to avoid HTML
# rendering engines from visually collapsing runs of spaces into a single
# space. We specifically don't replace single spaces for several reasons:
# 1. We'd break the HTML emitted by convert() by replacing spaces
# inside HTML elements (for example the spaces that separate
# element names from attribute names).
# 2. If every single space is replaced by a non-breaking space,
# web browsers perform awkwardly unintuitive word wrapping.
# 3. The HTML output would be bloated for no good reason.
text = re.sub(' {2,}', encode_whitespace_cb, text)
return text
def encode_whitespace_cb(match):
"""
Replace runs of multiple spaces with non-breaking spaces.
:param match: A regular expression match object.
:returns: The replacement string.
This function is used by func:`encode_whitespace()` as a callback for
replacement using a regular expression pattern.
"""
return '&nbsp;' * len(match.group(0))
def html_encode(text):
"""
Encode characters with a special meaning as HTML.
:param text: The plain text (a string).
:returns: The text converted to HTML (a string).
"""
text = text.replace('&', '&amp;')
text = text.replace('<', '&lt;')
text = text.replace('>', '&gt;')
text = text.replace('"', '&quot;')
return text
def parse_hex_color(value):
"""
Convert a CSS color in hexadecimal notation into its R, G, B components.
:param value: A CSS color in hexadecimal notation (a string like '#000000').
:return: A tuple with three integers (with values between 0 and 255)
corresponding to the R, G and B components of the color.
:raises: :exc:`~exceptions.ValueError` on values that can't be parsed.
"""
if value.startswith('#'):
value = value[1:]
if len(value) == 3:
return (
int(value[0] * 2, 16),
int(value[1] * 2, 16),
int(value[2] * 2, 16),
)
elif len(value) == 6:
return (
int(value[0:2], 16),
int(value[2:4], 16),
int(value[4:6], 16),
)
else:
raise ValueError()
def select_text_color(r, g, b):
"""
Choose a suitable color for the inverse text style.
:param r: The amount of red (an integer between 0 and 255).
:param g: The amount of green (an integer between 0 and 255).
:param b: The amount of blue (an integer between 0 and 255).
:returns: A CSS color in hexadecimal notation (a string).
In inverse mode the color that is normally used for the text is instead
used for the background, however this can render the text unreadable. The
purpose of :func:`select_text_color()` is to make an effort to select a
suitable text color. Based on http://stackoverflow.com/a/3943023/112731.
"""
return '#000' if (r * 0.299 + g * 0.587 + b * 0.114) > 186 else '#FFF'
class ColoredCronMailer(object):
"""
Easy to use integration between :mod:`coloredlogs` and the UNIX ``cron`` daemon.
By using :class:`ColoredCronMailer` as a context manager in the command
line interface of your Python program you make it trivially easy for users
of your program to opt in to HTML output under ``cron``: The only thing the
user needs to do is set ``CONTENT_TYPE="text/html"`` in their crontab!
Under the hood this requires quite a bit of magic and I must admit that I
developed this code simply because I was curious whether it could even be
done :-). It requires my :mod:`capturer` package which you can install
using ``pip install 'coloredlogs[cron]'``. The ``[cron]`` extra will pull
in the :mod:`capturer` 2.4 or newer which is required to capture the output
while silencing it - otherwise you'd get duplicate output in the emails
sent by ``cron``.
"""
def __init__(self):
"""Initialize output capturing when running under ``cron`` with the correct configuration."""
self.is_enabled = 'text/html' in os.environ.get('CONTENT_TYPE', 'text/plain')
self.is_silent = False
if self.is_enabled:
# We import capturer here so that the coloredlogs[cron] extra
# isn't required to use the other functions in this module.
from capturer import CaptureOutput
self.capturer = CaptureOutput(merged=True, relay=False)
def __enter__(self):
"""Start capturing output (when applicable)."""
if self.is_enabled:
self.capturer.__enter__()
return self
def __exit__(self, exc_type=None, exc_value=None, traceback=None):
"""Stop capturing output and convert the output to HTML (when applicable)."""
if self.is_enabled:
if not self.is_silent:
# Only call output() when we captured something useful.
text = self.capturer.get_text()
if text and not text.isspace():
output(convert(text))
self.capturer.__exit__(exc_type, exc_value, traceback)
def silence(self):
"""
Tell :func:`__exit__()` to swallow all output (things will be silent).
This can be useful when a Python program is written in such a way that
it has already produced output by the time it becomes apparent that
nothing useful can be done (say in a cron job that runs every few
minutes :-p). By calling :func:`silence()` the output can be swallowed
retroactively, avoiding useless emails from ``cron``.
"""
self.is_silent = True
@@ -0,0 +1,310 @@
# Mapping of ANSI color codes to HTML/CSS colors.
#
# Author: Peter Odding <peter@peterodding.com>
# Last Change: January 14, 2018
# URL: https://coloredlogs.readthedocs.io
"""Mapping of ANSI color codes to HTML/CSS colors."""
EIGHT_COLOR_PALETTE = (
'#010101', # black
'#DE382B', # red
'#39B54A', # green
'#FFC706', # yellow
'#006FB8', # blue
'#762671', # magenta
'#2CB5E9', # cyan
'#CCC', # white
)
"""
A tuple of strings mapping basic color codes to CSS colors.
The items in this tuple correspond to the eight basic color codes for black,
red, green, yellow, blue, magenta, cyan and white as defined in the original
standard for ANSI escape sequences. The CSS colors are based on the `Ubuntu
color scheme`_ described on Wikipedia and they are encoded as hexadecimal
values to get the shortest strings, which reduces the size (in bytes) of
conversion output.
.. _Ubuntu color scheme: https://en.wikipedia.org/wiki/ANSI_escape_code#Colors
"""
BRIGHT_COLOR_PALETTE = (
'#808080', # black
'#F00', # red
'#0F0', # green
'#FF0', # yellow
'#00F', # blue
'#F0F', # magenta
'#0FF', # cyan
'#FFF', # white
)
"""
A tuple of strings mapping bright color codes to CSS colors.
This tuple maps the bright color variants of :data:`EIGHT_COLOR_PALETTE`.
"""
EXTENDED_COLOR_PALETTE = (
'#000000',
'#800000',
'#008000',
'#808000',
'#000080',
'#800080',
'#008080',
'#C0C0C0',
'#808080',
'#FF0000',
'#00FF00',
'#FFFF00',
'#0000FF',
'#FF00FF',
'#00FFFF',
'#FFFFFF',
'#000000',
'#00005F',
'#000087',
'#0000AF',
'#0000D7',
'#0000FF',
'#005F00',
'#005F5F',
'#005F87',
'#005FAF',
'#005FD7',
'#005FFF',
'#008700',
'#00875F',
'#008787',
'#0087AF',
'#0087D7',
'#0087FF',
'#00AF00',
'#00AF5F',
'#00AF87',
'#00AFAF',
'#00AFD7',
'#00AFFF',
'#00D700',
'#00D75F',
'#00D787',
'#00D7AF',
'#00D7D7',
'#00D7FF',
'#00FF00',
'#00FF5F',
'#00FF87',
'#00FFAF',
'#00FFD7',
'#00FFFF',
'#5F0000',
'#5F005F',
'#5F0087',
'#5F00AF',
'#5F00D7',
'#5F00FF',
'#5F5F00',
'#5F5F5F',
'#5F5F87',
'#5F5FAF',
'#5F5FD7',
'#5F5FFF',
'#5F8700',
'#5F875F',
'#5F8787',
'#5F87AF',
'#5F87D7',
'#5F87FF',
'#5FAF00',
'#5FAF5F',
'#5FAF87',
'#5FAFAF',
'#5FAFD7',
'#5FAFFF',
'#5FD700',
'#5FD75F',
'#5FD787',
'#5FD7AF',
'#5FD7D7',
'#5FD7FF',
'#5FFF00',
'#5FFF5F',
'#5FFF87',
'#5FFFAF',
'#5FFFD7',
'#5FFFFF',
'#870000',
'#87005F',
'#870087',
'#8700AF',
'#8700D7',
'#8700FF',
'#875F00',
'#875F5F',
'#875F87',
'#875FAF',
'#875FD7',
'#875FFF',
'#878700',
'#87875F',
'#878787',
'#8787AF',
'#8787D7',
'#8787FF',
'#87AF00',
'#87AF5F',
'#87AF87',
'#87AFAF',
'#87AFD7',
'#87AFFF',
'#87D700',
'#87D75F',
'#87D787',
'#87D7AF',
'#87D7D7',
'#87D7FF',
'#87FF00',
'#87FF5F',
'#87FF87',
'#87FFAF',
'#87FFD7',
'#87FFFF',
'#AF0000',
'#AF005F',
'#AF0087',
'#AF00AF',
'#AF00D7',
'#AF00FF',
'#AF5F00',
'#AF5F5F',
'#AF5F87',
'#AF5FAF',
'#AF5FD7',
'#AF5FFF',
'#AF8700',
'#AF875F',
'#AF8787',
'#AF87AF',
'#AF87D7',
'#AF87FF',
'#AFAF00',
'#AFAF5F',
'#AFAF87',
'#AFAFAF',
'#AFAFD7',
'#AFAFFF',
'#AFD700',
'#AFD75F',
'#AFD787',
'#AFD7AF',
'#AFD7D7',
'#AFD7FF',
'#AFFF00',
'#AFFF5F',
'#AFFF87',
'#AFFFAF',
'#AFFFD7',
'#AFFFFF',
'#D70000',
'#D7005F',
'#D70087',
'#D700AF',
'#D700D7',
'#D700FF',
'#D75F00',
'#D75F5F',
'#D75F87',
'#D75FAF',
'#D75FD7',
'#D75FFF',
'#D78700',
'#D7875F',
'#D78787',
'#D787AF',
'#D787D7',
'#D787FF',
'#D7AF00',
'#D7AF5F',
'#D7AF87',
'#D7AFAF',
'#D7AFD7',
'#D7AFFF',
'#D7D700',
'#D7D75F',
'#D7D787',
'#D7D7AF',
'#D7D7D7',
'#D7D7FF',
'#D7FF00',
'#D7FF5F',
'#D7FF87',
'#D7FFAF',
'#D7FFD7',
'#D7FFFF',
'#FF0000',
'#FF005F',
'#FF0087',
'#FF00AF',
'#FF00D7',
'#FF00FF',
'#FF5F00',
'#FF5F5F',
'#FF5F87',
'#FF5FAF',
'#FF5FD7',
'#FF5FFF',
'#FF8700',
'#FF875F',
'#FF8787',
'#FF87AF',
'#FF87D7',
'#FF87FF',
'#FFAF00',
'#FFAF5F',
'#FFAF87',
'#FFAFAF',
'#FFAFD7',
'#FFAFFF',
'#FFD700',
'#FFD75F',
'#FFD787',
'#FFD7AF',
'#FFD7D7',
'#FFD7FF',
'#FFFF00',
'#FFFF5F',
'#FFFF87',
'#FFFFAF',
'#FFFFD7',
'#FFFFFF',
'#080808',
'#121212',
'#1C1C1C',
'#262626',
'#303030',
'#3A3A3A',
'#444444',
'#4E4E4E',
'#585858',
'#626262',
'#6C6C6C',
'#767676',
'#808080',
'#8A8A8A',
'#949494',
'#9E9E9E',
'#A8A8A8',
'#B2B2B2',
'#BCBCBC',
'#C6C6C6',
'#D0D0D0',
'#DADADA',
'#E4E4E4',
'#EEEEEE',
)
"""
A tuple of strings mapping 256 color mode color codes to CSS colors.
The items in this tuple correspond to the color codes in the 256 color mode palette.
"""
@@ -0,0 +1,49 @@
# Demonstration of the coloredlogs package.
#
# Author: Peter Odding <peter@peterodding.com>
# Last Change: January 14, 2018
# URL: https://coloredlogs.readthedocs.io
"""A simple demonstration of the `coloredlogs` package."""
# Standard library modules.
import os
import time
# Modules included in our package.
import coloredlogs
# If my verbose logger is installed, we'll use that for the demo.
try:
from verboselogs import VerboseLogger as getLogger
except ImportError:
from logging import getLogger
# Initialize a logger for this module.
logger = getLogger(__name__)
DEMO_DELAY = float(os.environ.get('COLOREDLOGS_DEMO_DELAY', '1'))
"""The number of seconds between each message emitted by :func:`demonstrate_colored_logging()`."""
def demonstrate_colored_logging():
"""Interactively demonstrate the :mod:`coloredlogs` package."""
# Determine the available logging levels and order them by numeric value.
decorated_levels = []
defined_levels = coloredlogs.find_defined_levels()
normalizer = coloredlogs.NameNormalizer()
for name, level in defined_levels.items():
if name != 'NOTSET':
item = (level, normalizer.normalize_name(name))
if item not in decorated_levels:
decorated_levels.append(item)
ordered_levels = sorted(decorated_levels)
# Initialize colored output to the terminal, default to the most
# verbose logging level but enable the user the customize it.
coloredlogs.install(level=os.environ.get('COLOREDLOGS_LOG_LEVEL', ordered_levels[0][1]))
# Print some examples with different timestamps.
for level, name in ordered_levels:
log_method = getattr(logger, name, None)
if log_method:
log_method("message with level %s (%i)", name, level)
time.sleep(DEMO_DELAY)
@@ -0,0 +1,292 @@
# Easy to use system logging for Python's logging module.
#
# Author: Peter Odding <peter@peterodding.com>
# Last Change: December 10, 2020
# URL: https://coloredlogs.readthedocs.io
"""
Easy to use UNIX system logging for Python's :mod:`logging` module.
Admittedly system logging has little to do with colored terminal output, however:
- The `coloredlogs` package is my attempt to do Python logging right and system
logging is an important part of that equation.
- I've seen a surprising number of quirks and mistakes in system logging done
in Python, for example including ``%(asctime)s`` in a format string (the
system logging daemon is responsible for adding timestamps and thus you end
up with duplicate timestamps that make the logs awful to read :-).
- The ``%(programname)s`` filter originated in my system logging code and I
wanted it in `coloredlogs` so the step to include this module wasn't that big.
- As a bonus this Python module now has a test suite and proper documentation.
So there :-P. Go take a look at :func:`enable_system_logging()`.
"""
# Standard library modules.
import logging
import logging.handlers
import os
import socket
import sys
# External dependencies.
from humanfriendly import coerce_boolean
from humanfriendly.compat import on_macos, on_windows
# Modules included in our package.
from coloredlogs import (
DEFAULT_LOG_LEVEL,
ProgramNameFilter,
adjust_level,
find_program_name,
level_to_number,
replace_handler,
)
LOG_DEVICE_MACOSX = '/var/run/syslog'
"""The pathname of the log device on Mac OS X (a string)."""
LOG_DEVICE_UNIX = '/dev/log'
"""The pathname of the log device on Linux and most other UNIX systems (a string)."""
DEFAULT_LOG_FORMAT = '%(programname)s[%(process)d]: %(levelname)s %(message)s'
"""
The default format for log messages sent to the system log (a string).
The ``%(programname)s`` format requires :class:`~coloredlogs.ProgramNameFilter`
but :func:`enable_system_logging()` takes care of this for you.
The ``name[pid]:`` construct (specifically the colon) in the format allows
rsyslogd_ to extract the ``$programname`` from each log message, which in turn
allows configuration files in ``/etc/rsyslog.d/*.conf`` to filter these log
messages to a separate log file (if the need arises).
.. _rsyslogd: https://en.wikipedia.org/wiki/Rsyslog
"""
# Initialize a logger for this module.
logger = logging.getLogger(__name__)
class SystemLogging(object):
"""Context manager to enable system logging."""
def __init__(self, *args, **kw):
"""
Initialize a :class:`SystemLogging` object.
:param args: Positional arguments to :func:`enable_system_logging()`.
:param kw: Keyword arguments to :func:`enable_system_logging()`.
"""
self.args = args
self.kw = kw
self.handler = None
def __enter__(self):
"""Enable system logging when entering the context."""
if self.handler is None:
self.handler = enable_system_logging(*self.args, **self.kw)
return self.handler
def __exit__(self, exc_type=None, exc_value=None, traceback=None):
"""
Disable system logging when leaving the context.
.. note:: If an exception is being handled when we leave the context a
warning message including traceback is logged *before* system
logging is disabled.
"""
if self.handler is not None:
if exc_type is not None:
logger.warning("Disabling system logging due to unhandled exception!", exc_info=True)
(self.kw.get('logger') or logging.getLogger()).removeHandler(self.handler)
self.handler = None
def enable_system_logging(programname=None, fmt=None, logger=None, reconfigure=True, **kw):
"""
Redirect :mod:`logging` messages to the system log (e.g. ``/var/log/syslog``).
:param programname: The program name to embed in log messages (a string, defaults
to the result of :func:`~coloredlogs.find_program_name()`).
:param fmt: The log format for system log messages (a string, defaults to
:data:`DEFAULT_LOG_FORMAT`).
:param logger: The logger to which the :class:`~logging.handlers.SysLogHandler`
should be connected (defaults to the root logger).
:param level: The logging level for the :class:`~logging.handlers.SysLogHandler`
(defaults to :data:`.DEFAULT_LOG_LEVEL`). This value is coerced
using :func:`~coloredlogs.level_to_number()`.
:param reconfigure: If :data:`True` (the default) multiple calls to
:func:`enable_system_logging()` will each override
the previous configuration.
:param kw: Refer to :func:`connect_to_syslog()`.
:returns: A :class:`~logging.handlers.SysLogHandler` object or
:data:`None`. If an existing handler is found and `reconfigure`
is :data:`False` the existing handler object is returned. If the
connection to the system logging daemon fails :data:`None` is
returned.
As of release 15.0 this function uses :func:`is_syslog_supported()` to
check whether system logging is supported and appropriate before it's
enabled.
.. note:: When the logger's effective level is too restrictive it is
relaxed (refer to `notes about log levels`_ for details).
"""
# Check whether system logging is supported / appropriate.
if not is_syslog_supported():
return None
# Provide defaults for omitted arguments.
programname = programname or find_program_name()
logger = logger or logging.getLogger()
fmt = fmt or DEFAULT_LOG_FORMAT
level = level_to_number(kw.get('level', DEFAULT_LOG_LEVEL))
# Check whether system logging is already enabled.
handler, logger = replace_handler(logger, match_syslog_handler, reconfigure)
# Make sure reconfiguration is allowed or not relevant.
if not (handler and not reconfigure):
# Create a system logging handler.
handler = connect_to_syslog(**kw)
# Make sure the handler was successfully created.
if handler:
# Enable the use of %(programname)s.
ProgramNameFilter.install(handler=handler, fmt=fmt, programname=programname)
# Connect the formatter, handler and logger.
handler.setFormatter(logging.Formatter(fmt))
logger.addHandler(handler)
# Adjust the level of the selected logger.
adjust_level(logger, level)
return handler
def connect_to_syslog(address=None, facility=None, level=None):
"""
Create a :class:`~logging.handlers.SysLogHandler`.
:param address: The device file or network address of the system logging
daemon (a string or tuple, defaults to the result of
:func:`find_syslog_address()`).
:param facility: Refer to :class:`~logging.handlers.SysLogHandler`.
Defaults to ``LOG_USER``.
:param level: The logging level for the :class:`~logging.handlers.SysLogHandler`
(defaults to :data:`.DEFAULT_LOG_LEVEL`). This value is coerced
using :func:`~coloredlogs.level_to_number()`.
:returns: A :class:`~logging.handlers.SysLogHandler` object or :data:`None` (if the
system logging daemon is unavailable).
The process of connecting to the system logging daemon goes as follows:
- The following two socket types are tried (in decreasing preference):
1. :data:`~socket.SOCK_RAW` avoids truncation of log messages but may
not be supported.
2. :data:`~socket.SOCK_STREAM` (TCP) supports longer messages than the
default (which is UDP).
"""
if not address:
address = find_syslog_address()
if facility is None:
facility = logging.handlers.SysLogHandler.LOG_USER
if level is None:
level = DEFAULT_LOG_LEVEL
for socktype in socket.SOCK_RAW, socket.SOCK_STREAM, None:
kw = dict(facility=facility, address=address)
if socktype is not None:
kw['socktype'] = socktype
try:
handler = logging.handlers.SysLogHandler(**kw)
except IOError:
# IOError is a superclass of socket.error which can be raised if the system
# logging daemon is unavailable.
pass
else:
handler.setLevel(level_to_number(level))
return handler
def find_syslog_address():
"""
Find the most suitable destination for system log messages.
:returns: The pathname of a log device (a string) or an address/port tuple as
supported by :class:`~logging.handlers.SysLogHandler`.
On Mac OS X this prefers :data:`LOG_DEVICE_MACOSX`, after that :data:`LOG_DEVICE_UNIX`
is checked for existence. If both of these device files don't exist the default used
by :class:`~logging.handlers.SysLogHandler` is returned.
"""
if sys.platform == 'darwin' and os.path.exists(LOG_DEVICE_MACOSX):
return LOG_DEVICE_MACOSX
elif os.path.exists(LOG_DEVICE_UNIX):
return LOG_DEVICE_UNIX
else:
return 'localhost', logging.handlers.SYSLOG_UDP_PORT
def is_syslog_supported():
"""
Determine whether system logging is supported.
:returns:
:data:`True` if system logging is supported and can be enabled,
:data:`False` if system logging is not supported or there are good
reasons for not enabling it.
The decision making process here is as follows:
Override
If the environment variable ``$COLOREDLOGS_SYSLOG`` is set it is evaluated
using :func:`~humanfriendly.coerce_boolean()` and the resulting value
overrides the platform detection discussed below, this allows users to
override the decision making process if they disagree / know better.
Linux / UNIX
On systems that are not Windows or MacOS (see below) we assume UNIX which
means either syslog is available or sending a bunch of UDP packets to
nowhere won't hurt anyone...
Microsoft Windows
Over the years I've had multiple reports of :pypi:`coloredlogs` spewing
extremely verbose errno 10057 warning messages to the console (once for
each log message I suppose) so I now assume it a default that
"syslog-style system logging" is not generally available on Windows.
Apple MacOS
There's cPython issue `#38780`_ which seems to result in a fatal exception
when the Python interpreter shuts down. This is (way) worse than not
having system logging enabled. The error message mentioned in `#38780`_
has actually been following me around for years now, see for example:
- https://github.com/xolox/python-rotate-backups/issues/9 mentions Docker
images implying Linux, so not strictly the same as `#38780`_.
- https://github.com/xolox/python-npm-accel/issues/4 is definitely related
to `#38780`_ and is what eventually prompted me to add the
:func:`is_syslog_supported()` logic.
.. _#38780: https://bugs.python.org/issue38780
"""
override = os.environ.get("COLOREDLOGS_SYSLOG")
if override is not None:
return coerce_boolean(override)
else:
return not (on_windows() or on_macos())
def match_syslog_handler(handler):
"""
Identify system logging handlers.
:param handler: The :class:`~logging.Handler` class to check.
:returns: :data:`True` if the handler is a
:class:`~logging.handlers.SysLogHandler`,
:data:`False` otherwise.
This function can be used as a callback for :func:`.find_handler()`.
"""
return isinstance(handler, logging.handlers.SysLogHandler)
@@ -0,0 +1,673 @@
# Automated tests for the `coloredlogs' package.
#
# Author: Peter Odding <peter@peterodding.com>
# Last Change: June 11, 2021
# URL: https://coloredlogs.readthedocs.io
"""Automated tests for the `coloredlogs` package."""
# Standard library modules.
import contextlib
import logging
import logging.handlers
import os
import re
import subprocess
import sys
import tempfile
# External dependencies.
from humanfriendly.compat import StringIO
from humanfriendly.terminal import ANSI_COLOR_CODES, ANSI_CSI, ansi_style, ansi_wrap
from humanfriendly.testing import PatchedAttribute, PatchedItem, TestCase, retry
from humanfriendly.text import format, random_string
# The module we're testing.
import coloredlogs
import coloredlogs.cli
from coloredlogs import (
CHROOT_FILES,
ColoredFormatter,
NameNormalizer,
decrease_verbosity,
find_defined_levels,
find_handler,
find_hostname,
find_program_name,
find_username,
get_level,
increase_verbosity,
install,
is_verbose,
level_to_number,
match_stream_handler,
parse_encoded_styles,
set_level,
walk_propagation_tree,
)
from coloredlogs.demo import demonstrate_colored_logging
from coloredlogs.syslog import SystemLogging, is_syslog_supported, match_syslog_handler
from coloredlogs.converter import (
ColoredCronMailer,
EIGHT_COLOR_PALETTE,
capture,
convert,
)
# External test dependencies.
from capturer import CaptureOutput
from verboselogs import VerboseLogger
# Compiled regular expression that matches a single line of output produced by
# the default log format (does not include matching of ANSI escape sequences).
PLAIN_TEXT_PATTERN = re.compile(r'''
(?P<date> \d{4}-\d{2}-\d{2} )
\s (?P<time> \d{2}:\d{2}:\d{2} )
\s (?P<hostname> \S+ )
\s (?P<logger_name> \w+ )
\[ (?P<process_id> \d+ ) \]
\s (?P<severity> [A-Z]+ )
\s (?P<message> .* )
''', re.VERBOSE)
# Compiled regular expression that matches a single line of output produced by
# the default log format with milliseconds=True.
PATTERN_INCLUDING_MILLISECONDS = re.compile(r'''
(?P<date> \d{4}-\d{2}-\d{2} )
\s (?P<time> \d{2}:\d{2}:\d{2},\d{3} )
\s (?P<hostname> \S+ )
\s (?P<logger_name> \w+ )
\[ (?P<process_id> \d+ ) \]
\s (?P<severity> [A-Z]+ )
\s (?P<message> .* )
''', re.VERBOSE)
def setUpModule():
"""Speed up the tests by disabling the demo's artificial delay."""
os.environ['COLOREDLOGS_DEMO_DELAY'] = '0'
coloredlogs.demo.DEMO_DELAY = 0
class ColoredLogsTestCase(TestCase):
"""Container for the `coloredlogs` tests."""
def find_system_log(self):
"""Find the system log file or skip the current test."""
filename = ('/var/log/system.log' if sys.platform == 'darwin' else (
'/var/log/syslog' if 'linux' in sys.platform else None
))
if not filename:
self.skipTest("Location of system log file unknown!")
elif not os.path.isfile(filename):
self.skipTest("System log file not found! (%s)" % filename)
elif not os.access(filename, os.R_OK):
self.skipTest("Insufficient permissions to read system log file! (%s)" % filename)
else:
return filename
def test_level_to_number(self):
"""Make sure :func:`level_to_number()` works as intended."""
# Make sure the default levels are translated as expected.
assert level_to_number('debug') == logging.DEBUG
assert level_to_number('info') == logging.INFO
assert level_to_number('warning') == logging.WARNING
assert level_to_number('error') == logging.ERROR
assert level_to_number('fatal') == logging.FATAL
# Make sure bogus level names don't blow up.
assert level_to_number('bogus-level') == logging.INFO
def test_find_hostname(self):
"""Make sure :func:`~find_hostname()` works correctly."""
assert find_hostname()
# Create a temporary file as a placeholder for e.g. /etc/debian_chroot.
fd, temporary_file = tempfile.mkstemp()
try:
with open(temporary_file, 'w') as handle:
handle.write('first line\n')
handle.write('second line\n')
CHROOT_FILES.insert(0, temporary_file)
# Make sure the chroot file is being read.
assert find_hostname() == 'first line'
finally:
# Clean up.
CHROOT_FILES.pop(0)
os.unlink(temporary_file)
# Test that unreadable chroot files don't break coloredlogs.
try:
CHROOT_FILES.insert(0, temporary_file)
# Make sure that a usable value is still produced.
assert find_hostname()
finally:
# Clean up.
CHROOT_FILES.pop(0)
def test_host_name_filter(self):
"""Make sure :func:`install()` integrates with :class:`~coloredlogs.HostNameFilter()`."""
install(fmt='%(hostname)s')
with CaptureOutput() as capturer:
logging.info("A truly insignificant message ..")
output = capturer.get_text()
assert find_hostname() in output
def test_program_name_filter(self):
"""Make sure :func:`install()` integrates with :class:`~coloredlogs.ProgramNameFilter()`."""
install(fmt='%(programname)s')
with CaptureOutput() as capturer:
logging.info("A truly insignificant message ..")
output = capturer.get_text()
assert find_program_name() in output
def test_username_filter(self):
"""Make sure :func:`install()` integrates with :class:`~coloredlogs.UserNameFilter()`."""
install(fmt='%(username)s')
with CaptureOutput() as capturer:
logging.info("A truly insignificant message ..")
output = capturer.get_text()
assert find_username() in output
def test_system_logging(self):
"""Make sure the :class:`coloredlogs.syslog.SystemLogging` context manager works."""
system_log_file = self.find_system_log()
expected_message = random_string(50)
with SystemLogging(programname='coloredlogs-test-suite') as syslog:
if not syslog:
return self.skipTest("couldn't connect to syslog daemon")
# When I tried out the system logging support on macOS 10.13.1 on
# 2018-01-05 I found that while WARNING and ERROR messages show up
# in the system log DEBUG and INFO messages don't. This explains
# the importance of the level of the log message below.
logging.error("%s", expected_message)
# Retry the following assertion (for up to 60 seconds) to give the
# logging daemon time to write our log message to disk. This
# appears to be needed on MacOS workers on Travis CI, see:
# https://travis-ci.org/xolox/python-coloredlogs/jobs/325245853
retry(lambda: check_contents(system_log_file, expected_message, True))
def test_system_logging_override(self):
"""Make sure the :class:`coloredlogs.syslog.is_syslog_supported` respects the override."""
with PatchedItem(os.environ, 'COLOREDLOGS_SYSLOG', 'true'):
assert is_syslog_supported() is True
with PatchedItem(os.environ, 'COLOREDLOGS_SYSLOG', 'false'):
assert is_syslog_supported() is False
def test_syslog_shortcut_simple(self):
"""Make sure that ``coloredlogs.install(syslog=True)`` works."""
system_log_file = self.find_system_log()
expected_message = random_string(50)
with cleanup_handlers():
# See test_system_logging() for the importance of this log level.
coloredlogs.install(syslog=True)
logging.error("%s", expected_message)
# See the comments in test_system_logging() on why this is retried.
retry(lambda: check_contents(system_log_file, expected_message, True))
def test_syslog_shortcut_enhanced(self):
"""Make sure that ``coloredlogs.install(syslog='warning')`` works."""
system_log_file = self.find_system_log()
the_expected_message = random_string(50)
not_an_expected_message = random_string(50)
with cleanup_handlers():
# See test_system_logging() for the importance of these log levels.
coloredlogs.install(syslog='error')
logging.warning("%s", not_an_expected_message)
logging.error("%s", the_expected_message)
# See the comments in test_system_logging() on why this is retried.
retry(lambda: check_contents(system_log_file, the_expected_message, True))
retry(lambda: check_contents(system_log_file, not_an_expected_message, False))
def test_name_normalization(self):
"""Make sure :class:`~coloredlogs.NameNormalizer` works as intended."""
nn = NameNormalizer()
for canonical_name in ['debug', 'info', 'warning', 'error', 'critical']:
assert nn.normalize_name(canonical_name) == canonical_name
assert nn.normalize_name(canonical_name.upper()) == canonical_name
assert nn.normalize_name('warn') == 'warning'
assert nn.normalize_name('fatal') == 'critical'
def test_style_parsing(self):
"""Make sure :func:`~coloredlogs.parse_encoded_styles()` works as intended."""
encoded_styles = 'debug=green;warning=yellow;error=red;critical=red,bold'
decoded_styles = parse_encoded_styles(encoded_styles, normalize_key=lambda k: k.upper())
assert sorted(decoded_styles.keys()) == sorted(['debug', 'warning', 'error', 'critical'])
assert decoded_styles['debug']['color'] == 'green'
assert decoded_styles['warning']['color'] == 'yellow'
assert decoded_styles['error']['color'] == 'red'
assert decoded_styles['critical']['color'] == 'red'
assert decoded_styles['critical']['bold'] is True
def test_is_verbose(self):
"""Make sure is_verbose() does what it should :-)."""
set_level(logging.INFO)
assert not is_verbose()
set_level(logging.DEBUG)
assert is_verbose()
set_level(logging.VERBOSE)
assert is_verbose()
def test_increase_verbosity(self):
"""Make sure increase_verbosity() respects default and custom levels."""
# Start from a known state.
set_level(logging.INFO)
assert get_level() == logging.INFO
# INFO -> VERBOSE.
increase_verbosity()
assert get_level() == logging.VERBOSE
# VERBOSE -> DEBUG.
increase_verbosity()
assert get_level() == logging.DEBUG
# DEBUG -> SPAM.
increase_verbosity()
assert get_level() == logging.SPAM
# SPAM -> NOTSET.
increase_verbosity()
assert get_level() == logging.NOTSET
# NOTSET -> NOTSET.
increase_verbosity()
assert get_level() == logging.NOTSET
def test_decrease_verbosity(self):
"""Make sure decrease_verbosity() respects default and custom levels."""
# Start from a known state.
set_level(logging.INFO)
assert get_level() == logging.INFO
# INFO -> NOTICE.
decrease_verbosity()
assert get_level() == logging.NOTICE
# NOTICE -> WARNING.
decrease_verbosity()
assert get_level() == logging.WARNING
# WARNING -> SUCCESS.
decrease_verbosity()
assert get_level() == logging.SUCCESS
# SUCCESS -> ERROR.
decrease_verbosity()
assert get_level() == logging.ERROR
# ERROR -> CRITICAL.
decrease_verbosity()
assert get_level() == logging.CRITICAL
# CRITICAL -> CRITICAL.
decrease_verbosity()
assert get_level() == logging.CRITICAL
def test_level_discovery(self):
"""Make sure find_defined_levels() always reports the levels defined in Python's standard library."""
defined_levels = find_defined_levels()
level_values = defined_levels.values()
for number in (0, 10, 20, 30, 40, 50):
assert number in level_values
def test_walk_propagation_tree(self):
"""Make sure walk_propagation_tree() properly walks the tree of loggers."""
root, parent, child, grand_child = self.get_logger_tree()
# Check the default mode of operation.
loggers = list(walk_propagation_tree(grand_child))
assert loggers == [grand_child, child, parent, root]
# Now change the propagation (non-default mode of operation).
child.propagate = False
loggers = list(walk_propagation_tree(grand_child))
assert loggers == [grand_child, child]
def test_find_handler(self):
"""Make sure find_handler() works as intended."""
root, parent, child, grand_child = self.get_logger_tree()
# Add some handlers to the tree.
stream_handler = logging.StreamHandler()
syslog_handler = logging.handlers.SysLogHandler()
child.addHandler(stream_handler)
parent.addHandler(syslog_handler)
# Make sure the first matching handler is returned.
matched_handler, matched_logger = find_handler(grand_child, lambda h: isinstance(h, logging.Handler))
assert matched_handler is stream_handler
# Make sure the first matching handler of the given type is returned.
matched_handler, matched_logger = find_handler(child, lambda h: isinstance(h, logging.handlers.SysLogHandler))
assert matched_handler is syslog_handler
def get_logger_tree(self):
"""Create and return a tree of loggers."""
# Get the root logger.
root = logging.getLogger()
# Create a top level logger for ourselves.
parent_name = random_string()
parent = logging.getLogger(parent_name)
# Create a child logger.
child_name = '%s.%s' % (parent_name, random_string())
child = logging.getLogger(child_name)
# Create a grand child logger.
grand_child_name = '%s.%s' % (child_name, random_string())
grand_child = logging.getLogger(grand_child_name)
return root, parent, child, grand_child
def test_support_for_milliseconds(self):
"""Make sure milliseconds are hidden by default but can be easily enabled."""
# Check that the default log format doesn't include milliseconds.
stream = StringIO()
install(reconfigure=True, stream=stream)
logging.info("This should not include milliseconds.")
assert all(map(PLAIN_TEXT_PATTERN.match, stream.getvalue().splitlines()))
# Check that milliseconds can be enabled via a shortcut.
stream = StringIO()
install(milliseconds=True, reconfigure=True, stream=stream)
logging.info("This should include milliseconds.")
assert all(map(PATTERN_INCLUDING_MILLISECONDS.match, stream.getvalue().splitlines()))
def test_support_for_milliseconds_directive(self):
"""Make sure milliseconds using the ``%f`` directive are supported."""
stream = StringIO()
install(reconfigure=True, stream=stream, datefmt='%Y-%m-%dT%H:%M:%S.%f%z')
logging.info("This should be timestamped according to #45.")
assert re.match(r'^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}[+-]\d{4}\s', stream.getvalue())
def test_plain_text_output_format(self):
"""Inspect the plain text output of coloredlogs."""
logger = VerboseLogger(random_string(25))
stream = StringIO()
install(level=logging.NOTSET, logger=logger, stream=stream)
# Test that filtering on severity works.
logger.setLevel(logging.INFO)
logger.debug("No one should see this message.")
assert len(stream.getvalue().strip()) == 0
# Test that the default output format looks okay in plain text.
logger.setLevel(logging.NOTSET)
for method, severity in ((logger.debug, 'DEBUG'),
(logger.info, 'INFO'),
(logger.verbose, 'VERBOSE'),
(logger.warning, 'WARNING'),
(logger.error, 'ERROR'),
(logger.critical, 'CRITICAL')):
# XXX Workaround for a regression in Python 3.7 caused by the
# Logger.isEnabledFor() method using stale cache entries. If we
# don't clear the cache then logger.isEnabledFor(logging.DEBUG)
# returns False and no DEBUG message is emitted.
try:
logger._cache.clear()
except AttributeError:
pass
# Prepare the text.
text = "This is a message with severity %r." % severity.lower()
# Log the message with the given severity.
method(text)
# Get the line of output generated by the handler.
output = stream.getvalue()
lines = output.splitlines()
last_line = lines[-1]
assert text in last_line
assert severity in last_line
assert PLAIN_TEXT_PATTERN.match(last_line)
def test_dynamic_stderr_lookup(self):
"""Make sure coloredlogs.install() uses StandardErrorHandler when possible."""
coloredlogs.install()
# Redirect sys.stderr to a temporary buffer.
initial_stream = StringIO()
initial_text = "Which stream will receive this text?"
with PatchedAttribute(sys, 'stderr', initial_stream):
logging.info(initial_text)
assert initial_text in initial_stream.getvalue()
# Redirect sys.stderr again, to a different destination.
subsequent_stream = StringIO()
subsequent_text = "And which stream will receive this other text?"
with PatchedAttribute(sys, 'stderr', subsequent_stream):
logging.info(subsequent_text)
assert subsequent_text in subsequent_stream.getvalue()
def test_force_enable(self):
"""Make sure ANSI escape sequences can be forced (bypassing auto-detection)."""
interpreter = subprocess.Popen([
sys.executable, "-c", ";".join([
"import coloredlogs, logging",
"coloredlogs.install(isatty=True)",
"logging.info('Hello world')",
]),
], stderr=subprocess.PIPE)
stdout, stderr = interpreter.communicate()
assert ANSI_CSI in stderr.decode('UTF-8')
def test_auto_disable(self):
"""
Make sure ANSI escape sequences are not emitted when logging output is being redirected.
This is a regression test for https://github.com/xolox/python-coloredlogs/issues/100.
It works as follows:
1. We mock an interactive terminal using 'capturer' to ensure that this
test works inside test drivers that capture output (like pytest).
2. We launch a subprocess (to ensure a clean process state) where
stderr is captured but stdout is not, emulating issue #100.
3. The output captured on stderr contained ANSI escape sequences after
this test was written and before the issue was fixed, so now this
serves as a regression test for issue #100.
"""
with CaptureOutput():
interpreter = subprocess.Popen([
sys.executable, "-c", ";".join([
"import coloredlogs, logging",
"coloredlogs.install()",
"logging.info('Hello world')",
]),
], stderr=subprocess.PIPE)
stdout, stderr = interpreter.communicate()
assert ANSI_CSI not in stderr.decode('UTF-8')
def test_env_disable(self):
"""Make sure ANSI escape sequences can be disabled using ``$NO_COLOR``."""
with PatchedItem(os.environ, 'NO_COLOR', 'I like monochrome'):
with CaptureOutput() as capturer:
subprocess.check_call([
sys.executable, "-c", ";".join([
"import coloredlogs, logging",
"coloredlogs.install()",
"logging.info('Hello world')",
]),
])
output = capturer.get_text()
assert ANSI_CSI not in output
def test_html_conversion(self):
"""Check the conversion from ANSI escape sequences to HTML."""
# Check conversion of colored text.
for color_name, ansi_code in ANSI_COLOR_CODES.items():
ansi_encoded_text = 'plain text followed by %s text' % ansi_wrap(color_name, color=color_name)
expected_html = format(
'<code>plain text followed by <span style="color:{css}">{name}</span> text</code>',
css=EIGHT_COLOR_PALETTE[ansi_code], name=color_name,
)
self.assertEqual(expected_html, convert(ansi_encoded_text))
# Check conversion of bright colored text.
expected_html = '<code><span style="color:#FF0">bright yellow</span></code>'
self.assertEqual(expected_html, convert(ansi_wrap('bright yellow', color='yellow', bright=True)))
# Check conversion of text with a background color.
expected_html = '<code><span style="background-color:#DE382B">red background</span></code>'
self.assertEqual(expected_html, convert(ansi_wrap('red background', background='red')))
# Check conversion of text with a bright background color.
expected_html = '<code><span style="background-color:#F00">bright red background</span></code>'
self.assertEqual(expected_html, convert(ansi_wrap('bright red background', background='red', bright=True)))
# Check conversion of text that uses the 256 color mode palette as a foreground color.
expected_html = '<code><span style="color:#FFAF00">256 color mode foreground</span></code>'
self.assertEqual(expected_html, convert(ansi_wrap('256 color mode foreground', color=214)))
# Check conversion of text that uses the 256 color mode palette as a background color.
expected_html = '<code><span style="background-color:#AF0000">256 color mode background</span></code>'
self.assertEqual(expected_html, convert(ansi_wrap('256 color mode background', background=124)))
# Check that invalid 256 color mode indexes don't raise exceptions.
expected_html = '<code>plain text expected</code>'
self.assertEqual(expected_html, convert('\x1b[38;5;256mplain text expected\x1b[0m'))
# Check conversion of bold text.
expected_html = '<code><span style="font-weight:bold">bold text</span></code>'
self.assertEqual(expected_html, convert(ansi_wrap('bold text', bold=True)))
# Check conversion of underlined text.
expected_html = '<code><span style="text-decoration:underline">underlined text</span></code>'
self.assertEqual(expected_html, convert(ansi_wrap('underlined text', underline=True)))
# Check conversion of strike-through text.
expected_html = '<code><span style="text-decoration:line-through">strike-through text</span></code>'
self.assertEqual(expected_html, convert(ansi_wrap('strike-through text', strike_through=True)))
# Check conversion of inverse text.
expected_html = '<code><span style="background-color:#FFC706;color:#000">inverse</span></code>'
self.assertEqual(expected_html, convert(ansi_wrap('inverse', color='yellow', inverse=True)))
# Check conversion of URLs.
for sample_text in 'www.python.org', 'http://coloredlogs.rtfd.org', 'https://coloredlogs.rtfd.org':
sample_url = sample_text if '://' in sample_text else ('http://' + sample_text)
expected_html = '<code><a href="%s" style="color:inherit">%s</a></code>' % (sample_url, sample_text)
self.assertEqual(expected_html, convert(sample_text))
# Check that the capture pattern for URLs doesn't match ANSI escape
# sequences and also check that the short hand for the 0 reset code is
# supported. These are tests for regressions of bugs found in
# coloredlogs <= 8.0.
reset_short_hand = '\x1b[0m'
blue_underlined = ansi_style(color='blue', underline=True)
ansi_encoded_text = '<%shttps://coloredlogs.readthedocs.io%s>' % (blue_underlined, reset_short_hand)
expected_html = (
'<code>&lt;<span style="color:#006FB8;text-decoration:underline">'
'<a href="https://coloredlogs.readthedocs.io" style="color:inherit">'
'https://coloredlogs.readthedocs.io'
'</a></span>&gt;</code>'
)
self.assertEqual(expected_html, convert(ansi_encoded_text))
def test_output_interception(self):
"""Test capturing of output from external commands."""
expected_output = 'testing, 1, 2, 3 ..'
actual_output = capture(['echo', expected_output])
assert actual_output.strip() == expected_output.strip()
def test_enable_colored_cron_mailer(self):
"""Test that automatic ANSI to HTML conversion when running under ``cron`` can be enabled."""
with PatchedItem(os.environ, 'CONTENT_TYPE', 'text/html'):
with ColoredCronMailer() as mailer:
assert mailer.is_enabled
def test_disable_colored_cron_mailer(self):
"""Test that automatic ANSI to HTML conversion when running under ``cron`` can be disabled."""
with PatchedItem(os.environ, 'CONTENT_TYPE', 'text/plain'):
with ColoredCronMailer() as mailer:
assert not mailer.is_enabled
def test_auto_install(self):
"""Test :func:`coloredlogs.auto_install()`."""
needle = random_string()
command_line = [sys.executable, '-c', 'import logging; logging.info(%r)' % needle]
# Sanity check that log messages aren't enabled by default.
with CaptureOutput() as capturer:
os.environ['COLOREDLOGS_AUTO_INSTALL'] = 'false'
subprocess.check_call(command_line)
output = capturer.get_text()
assert needle not in output
# Test that the $COLOREDLOGS_AUTO_INSTALL environment variable can be
# used to automatically call coloredlogs.install() during initialization.
with CaptureOutput() as capturer:
os.environ['COLOREDLOGS_AUTO_INSTALL'] = 'true'
subprocess.check_call(command_line)
output = capturer.get_text()
assert needle in output
def test_cli_demo(self):
"""Test the command line colored logging demonstration."""
with CaptureOutput() as capturer:
main('coloredlogs', '--demo')
output = capturer.get_text()
# Make sure the output contains all of the expected logging level names.
for name in 'debug', 'info', 'warning', 'error', 'critical':
assert name.upper() in output
def test_cli_conversion(self):
"""Test the command line HTML conversion."""
output = main('coloredlogs', '--convert', 'coloredlogs', '--demo', capture=True)
# Make sure the output is encoded as HTML.
assert '<span' in output
def test_empty_conversion(self):
"""
Test that conversion of empty output produces no HTML.
This test was added because I found that ``coloredlogs --convert`` when
used in a cron job could cause cron to send out what appeared to be
empty emails. On more careful inspection the body of those emails was
``<code></code>``. By not emitting the wrapper element when no other
HTML is generated, cron will not send out an email.
"""
output = main('coloredlogs', '--convert', 'true', capture=True)
assert not output.strip()
def test_implicit_usage_message(self):
"""Test that the usage message is shown when no actions are given."""
assert 'Usage:' in main('coloredlogs', capture=True)
def test_explicit_usage_message(self):
"""Test that the usage message is shown when ``--help`` is given."""
assert 'Usage:' in main('coloredlogs', '--help', capture=True)
def test_custom_record_factory(self):
"""
Test that custom LogRecord factories are supported.
This test is a bit convoluted because the logging module suppresses
exceptions. We monkey patch the method suspected of encountering
exceptions so that we can tell after it was called whether any
exceptions occurred (despite the exceptions not propagating).
"""
if not hasattr(logging, 'getLogRecordFactory'):
return self.skipTest("this test requires Python >= 3.2")
exceptions = []
original_method = ColoredFormatter.format
original_factory = logging.getLogRecordFactory()
def custom_factory(*args, **kwargs):
record = original_factory(*args, **kwargs)
record.custom_attribute = 0xdecafbad
return record
def custom_method(*args, **kw):
try:
return original_method(*args, **kw)
except Exception as e:
exceptions.append(e)
raise
with PatchedAttribute(ColoredFormatter, 'format', custom_method):
logging.setLogRecordFactory(custom_factory)
try:
demonstrate_colored_logging()
finally:
logging.setLogRecordFactory(original_factory)
# Ensure that no exceptions were triggered.
assert not exceptions
def check_contents(filename, contents, match):
"""Check if a line in a file contains an expected string."""
with open(filename) as handle:
assert any(contents in line for line in handle) == match
def main(*arguments, **options):
"""Wrap the command line interface to make it easier to test."""
capture = options.get('capture', False)
saved_argv = sys.argv
saved_stdout = sys.stdout
try:
sys.argv = arguments
if capture:
sys.stdout = StringIO()
coloredlogs.cli.main()
if capture:
return sys.stdout.getvalue()
finally:
sys.argv = saved_argv
sys.stdout = saved_stdout
@contextlib.contextmanager
def cleanup_handlers():
"""Context manager to cleanup output handlers."""
# There's nothing to set up so we immediately yield control.
yield
# After the with block ends we cleanup any output handlers.
for match_func in match_stream_handler, match_syslog_handler:
handler, logger = find_handler(logging.getLogger(), match_func)
if handler and logger:
logger.removeHandler(handler)
@@ -0,0 +1,27 @@
Metadata-Version: 2.4
Name: flatbuffers
Version: 25.12.19
Summary: The FlatBuffers serialization format for Python
Home-page: https://google.github.io/flatbuffers/
Author: Derek Bailey
Author-email: derekbailey@google.com
License: Apache 2.0
Project-URL: Documentation, https://google.github.io/flatbuffers/
Project-URL: Source, https://github.com/google/flatbuffers
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: home-page
Dynamic: license
Dynamic: project-url
Dynamic: summary
Python runtime library for use with the `Flatbuffers <https://google.github.io/flatbuffers/>`_ serialization format.
@@ -0,0 +1,25 @@
flatbuffers-25.12.19.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
flatbuffers-25.12.19.dist-info/METADATA,sha256=nzgjn5b6ohUipqratpEuHlYHCLvpyfaWoGeclAP6CJ4,1004
flatbuffers-25.12.19.dist-info/RECORD,,
flatbuffers-25.12.19.dist-info/WHEEL,sha256=JNWh1Fm1UdwIQV075glCn4MVuCRs0sotJIq-J6rbxCU,109
flatbuffers-25.12.19.dist-info/top_level.txt,sha256=UXVWLA8ys6HeqTz6rfKesocUq6ln-ZL8mhZC_cq5BEc,12
flatbuffers/__init__.py,sha256=vJZrqZOOTKdBNMa_iTKUA6WJG_c_NzKGpFXOe1Igtiw,751
flatbuffers/__pycache__/__init__.cpython-312.pyc,,
flatbuffers/__pycache__/_version.cpython-312.pyc,,
flatbuffers/__pycache__/builder.cpython-312.pyc,,
flatbuffers/__pycache__/compat.cpython-312.pyc,,
flatbuffers/__pycache__/encode.cpython-312.pyc,,
flatbuffers/__pycache__/flexbuffers.cpython-312.pyc,,
flatbuffers/__pycache__/number_types.cpython-312.pyc,,
flatbuffers/__pycache__/packer.cpython-312.pyc,,
flatbuffers/__pycache__/table.cpython-312.pyc,,
flatbuffers/__pycache__/util.cpython-312.pyc,,
flatbuffers/_version.py,sha256=sk31rbYWDseoJxI9YQ1fYR-MMxRATyqsntKAEv7D7pI,696
flatbuffers/builder.py,sha256=HrG5KJ9rasiSTrMGeatkSdDs7fXV5fy_927Dsgakp4A,24567
flatbuffers/compat.py,sha256=ihBSpWDCSL-vgLSyZtcu8LX3ZI3wz9LhtqItY2GQZgg,2373
flatbuffers/encode.py,sha256=2Or3mgWRAkJiWg-GgYasDU4zIHpQU3W06fmIhwbz5uM,1550
flatbuffers/flexbuffers.py,sha256=yF8Wr4Lo8WJb-pj9NNaIYxLwzlHHyTroM0iO8fyDwbU,44454
flatbuffers/number_types.py,sha256=ijO0QcJiuxlQegoBOed0v9m0DdzTZHWxpTBZUqzsWHA,3762
flatbuffers/packer.py,sha256=LNWym8YgFRqHjcPeGpYY3inCGWH6XnbkQKtAPtFEVas,1164
flatbuffers/table.py,sha256=ciYTmq_CzAuYpb3KAVnl75M84ieChfbyKne-dFHzwwU,4818
flatbuffers/util.py,sha256=mRVQ1VoHp0MJMNtRTUGVzALwN4T_C-U14tMbj99py2A,1608
@@ -0,0 +1,6 @@
Wheel-Version: 1.0
Generator: setuptools (80.9.0)
Root-Is-Purelib: true
Tag: py2-none-any
Tag: py3-none-any
@@ -0,0 +1,19 @@
# Copyright 2014 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from . import util
from ._version import __version__
from .builder import Builder
from .compat import range_func as compat_range
from .table import Table
@@ -0,0 +1,17 @@
# Copyright 2019 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Placeholder, to be updated during the release process
# by the setup.py
__version__ = "25.12.19"
@@ -0,0 +1,858 @@
# Copyright 2014 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import warnings
from . import compat
from . import encode
from . import number_types as N
from . import packer
from .compat import memoryview_type
from .compat import NumpyRequiredForThisFeature, import_numpy
from .compat import range_func
from .number_types import (SOffsetTFlags, UOffsetTFlags, VOffsetTFlags)
np = import_numpy()
## @file
## @addtogroup flatbuffers_python_api
## @{
## @cond FLATBUFFERS_INTERNAL
class OffsetArithmeticError(RuntimeError):
"""Error caused by an Offset arithmetic error.
Probably caused by bad writing of fields. This is considered an unreachable
situation in normal circumstances.
"""
pass
class IsNotNestedError(RuntimeError):
"""Error caused by using a Builder to write Object data when not inside
an Object.
"""
pass
class IsNestedError(RuntimeError):
"""Error caused by using a Builder to begin an Object when an Object is
already being built.
"""
pass
class StructIsNotInlineError(RuntimeError):
"""Error caused by using a Builder to write a Struct at a location that
is not the current Offset.
"""
pass
class BuilderSizeError(RuntimeError):
"""Error caused by causing a Builder to exceed the hardcoded limit of 2
gigabytes.
"""
pass
class BuilderNotFinishedError(RuntimeError):
"""Error caused by not calling `Finish` before calling `Output`."""
pass
class EndVectorLengthMismatched(RuntimeError):
"""The number of elements passed to EndVector does not match the number
specified in StartVector.
"""
pass
# VtableMetadataFields is the count of metadata fields in each vtable.
VtableMetadataFields = 2
## @endcond
class Builder(object):
"""A Builder is used to construct one or more FlatBuffers.
Typically, Builder objects will be used from code generated by the `flatc`
compiler.
A Builder constructs byte buffers in a last-first manner for simplicity and
performance during reading.
Internally, a Builder is a state machine for creating FlatBuffer objects.
It holds the following internal state:
- Bytes: an array of bytes.
- current_vtable: a list of integers.
- vtables: a hash of vtable entries.
Attributes:
Bytes: The internal `bytearray` for the Builder.
finished: A boolean determining if the Builder has been finalized.
"""
## @cond FLATBUFFERS_INTENRAL
__slots__ = (
"Bytes",
"current_vtable",
"head",
"minalign",
"objectEnd",
"vtables",
"nested",
"forceDefaults",
"finished",
"vectorNumElems",
"sharedStrings",
)
"""Maximum buffer size constant, in bytes.
Builder will never allow it's buffer grow over this size.
Currently equals 2Gb.
"""
MAX_BUFFER_SIZE = 2**31
## @endcond
def __init__(self, initialSize=1024):
"""Initializes a Builder of size `initial_size`.
The internal buffer is grown as needed.
"""
if not (0 <= initialSize <= Builder.MAX_BUFFER_SIZE):
msg = "flatbuffers: Cannot create Builder larger than 2 gigabytes."
raise BuilderSizeError(msg)
self.Bytes = bytearray(initialSize)
## @cond FLATBUFFERS_INTERNAL
self.current_vtable = None
self.head = UOffsetTFlags.py_type(initialSize)
self.minalign = 1
self.objectEnd = None
self.vtables = {}
self.nested = False
self.forceDefaults = False
self.sharedStrings = None
## @endcond
self.finished = False
def Clear(self):
## @cond FLATBUFFERS_INTERNAL
self.current_vtable = None
self.head = len(self.Bytes)
self.minalign = 1
self.objectEnd = None
self.vtables = {}
self.nested = False
self.forceDefaults = False
self.sharedStrings = None
self.vectorNumElems = None
## @endcond
self.finished = False
def Output(self):
"""Return the portion of the buffer that has been used for writing data.
This is the typical way to access the FlatBuffer data inside the
builder. If you try to access `Builder.Bytes` directly, you would need
to manually index it with `Head()`, since the buffer is constructed
backwards.
It raises BuilderNotFinishedError if the buffer has not been finished
with `Finish`.
"""
if not self.finished:
raise BuilderNotFinishedError()
return self.Bytes[self.head :]
## @cond FLATBUFFERS_INTERNAL
def StartObject(self, numfields):
"""StartObject initializes bookkeeping for writing a new object."""
self.assertNotNested()
# use 32-bit offsets so that arithmetic doesn't overflow.
self.current_vtable = [0] * numfields
self.objectEnd = self.Offset()
self.nested = True
def WriteVtable(self):
"""WriteVtable serializes the vtable for the current object, if needed.
Before writing out the vtable, this checks pre-existing vtables for
equality to this one. If an equal vtable is found, point the object to
the existing vtable and return.
Because vtable values are sensitive to alignment of object data, not
all logically-equal vtables will be deduplicated.
A vtable has the following format:
<VOffsetT: size of the vtable in bytes, including this value>
<VOffsetT: size of the object in bytes, including the vtable offset>
<VOffsetT: offset for a field> * N, where N is the number of fields
in the schema for this type. Includes deprecated fields.
Thus, a vtable is made of 2 + N elements, each VOffsetT bytes wide.
An object has the following format:
<SOffsetT: offset to this object's vtable (may be negative)>
<byte: data>+
"""
# Prepend a zero scalar to the object. Later in this function we'll
# write an offset here that points to the object's vtable:
self.PrependSOffsetTRelative(0)
objectOffset = self.Offset()
vtKey = []
trim = True
for elem in reversed(self.current_vtable):
if elem == 0:
if trim:
continue
else:
elem = objectOffset - elem
trim = False
vtKey.append(elem)
objectSize = UOffsetTFlags.py_type(objectOffset - self.objectEnd)
vtKey.append(objectSize)
vtKey = tuple(vtKey)
# calculate the size of the object
vt2Offset = self.vtables.get(vtKey)
if vt2Offset is None:
# Did not find a vtable, so write this one to the buffer.
# Write out the current vtable in reverse , because
# serialization occurs in last-first order:
i = len(self.current_vtable) - 1
trailing = 0
trim = True
while i >= 0:
off = 0
elem = self.current_vtable[i]
i -= 1
if elem == 0:
if trim:
trailing += 1
continue
else:
# Forward reference to field;
# use 32bit number to ensure no overflow:
off = objectOffset - elem
trim = False
self.PrependVOffsetT(off)
# The two metadata fields are written last.
# First, store the object bytesize:
self.PrependVOffsetT(VOffsetTFlags.py_type(objectSize))
# Second, store the vtable bytesize:
vBytes = len(self.current_vtable) - trailing + VtableMetadataFields
vBytes *= N.VOffsetTFlags.bytewidth
self.PrependVOffsetT(VOffsetTFlags.py_type(vBytes))
# Next, write the offset to the new vtable in the
# already-allocated SOffsetT at the beginning of this object:
objectStart = SOffsetTFlags.py_type(len(self.Bytes) - objectOffset)
encode.Write(
packer.soffset,
self.Bytes,
objectStart,
SOffsetTFlags.py_type(self.Offset() - objectOffset),
)
# Finally, store this vtable in memory for future
# deduplication:
self.vtables[vtKey] = self.Offset()
else:
# Found a duplicate vtable.
objectStart = SOffsetTFlags.py_type(len(self.Bytes) - objectOffset)
self.head = UOffsetTFlags.py_type(objectStart)
# Write the offset to the found vtable in the
# already-allocated SOffsetT at the beginning of this object:
encode.Write(
packer.soffset,
self.Bytes,
self.Head(),
SOffsetTFlags.py_type(vt2Offset - objectOffset),
)
self.current_vtable = None
return objectOffset
def EndObject(self):
"""EndObject writes data necessary to finish object construction."""
self.assertNested()
self.nested = False
return self.WriteVtable()
def GrowByteBuffer(self):
"""Doubles the size of the byteslice, and copies the old data towards
the end of the new buffer (since we build the buffer backwards).
"""
if len(self.Bytes) == Builder.MAX_BUFFER_SIZE:
msg = "flatbuffers: cannot grow buffer beyond 2 gigabytes"
raise BuilderSizeError(msg)
newSize = min(len(self.Bytes) * 2, Builder.MAX_BUFFER_SIZE)
if newSize == 0:
newSize = 1
bytes2 = bytearray(newSize)
bytes2[newSize - len(self.Bytes) :] = self.Bytes
self.Bytes = bytes2
## @endcond
def Head(self):
"""Get the start of useful data in the underlying byte buffer.
Note: unlike other functions, this value is interpreted as from the
left.
"""
## @cond FLATBUFFERS_INTERNAL
return self.head
## @endcond
## @cond FLATBUFFERS_INTERNAL
def Offset(self):
"""Offset relative to the end of the buffer."""
return len(self.Bytes) - self.head
def Pad(self, n):
"""Pad places zeros at the current offset."""
if n <= 0:
return
new_head = self.head - n
self.Bytes[new_head : self.head] = b"\x00" * n
self.head = new_head
def Prep(self, size, additionalBytes):
"""Prep prepares to write an element of `size` after `additional_bytes`
have been written, e.g. if you write a string, you need to align
such the int length field is aligned to SizeInt32, and the string
data follows it directly.
If all you need to do is align, `additionalBytes` will be 0.
"""
# Track the biggest thing we've ever aligned to.
if size > self.minalign:
self.minalign = size
# Find the amount of alignment needed such that `size` is properly
# aligned after `additionalBytes`:
head = self.head
buf_len = len(self.Bytes)
alignSize = (~(buf_len - head + additionalBytes)) + 1
alignSize &= size - 1
# Reallocate the buffer if needed:
needed = alignSize + size + additionalBytes
while head < needed:
oldBufSize = buf_len
self.GrowByteBuffer()
buf_len = len(self.Bytes)
head += buf_len - oldBufSize
self.head = head
self.Pad(alignSize)
def PrependSOffsetTRelative(self, off):
"""PrependSOffsetTRelative prepends an SOffsetT, relative to where it
will be written.
"""
# Ensure alignment is already done:
self.Prep(N.SOffsetTFlags.bytewidth, 0)
if not (off <= self.Offset()):
msg = "flatbuffers: Offset arithmetic error."
raise OffsetArithmeticError(msg)
off2 = self.Offset() - off + N.SOffsetTFlags.bytewidth
self.PlaceSOffsetT(off2)
## @endcond
def PrependUOffsetTRelative(self, off):
"""Prepends an unsigned offset into vector data, relative to where it
will be written.
"""
# Ensure alignment is already done:
self.Prep(N.UOffsetTFlags.bytewidth, 0)
if not (off <= self.Offset()):
msg = "flatbuffers: Offset arithmetic error."
raise OffsetArithmeticError(msg)
off2 = self.Offset() - off + N.UOffsetTFlags.bytewidth
self.PlaceUOffsetT(off2)
## @cond FLATBUFFERS_INTERNAL
def StartVector(self, elemSize, numElems, alignment):
"""StartVector initializes bookkeeping for writing a new vector.
A vector has the following format:
- <UOffsetT: number of elements in this vector>
- <T: data>+, where T is the type of elements of this vector.
"""
self.assertNotNested()
self.nested = True
self.vectorNumElems = numElems
self.Prep(N.Uint32Flags.bytewidth, elemSize * numElems)
self.Prep(alignment, elemSize * numElems) # In case alignment > int.
return self.Offset()
## @endcond
def EndVector(self, numElems=None):
"""EndVector writes data necessary to finish vector construction."""
self.assertNested()
## @cond FLATBUFFERS_INTERNAL
self.nested = False
## @endcond
if numElems:
warnings.warn("numElems is deprecated.", DeprecationWarning, stacklevel=2)
if numElems != self.vectorNumElems:
raise EndVectorLengthMismatched()
# we already made space for this, so write without PrependUint32
self.PlaceUOffsetT(self.vectorNumElems)
self.vectorNumElems = None
return self.Offset()
def CreateSharedString(self, s, encoding="utf-8", errors="strict"):
"""CreateSharedString checks if the string is already written to the buffer
before calling CreateString.
"""
if not self.sharedStrings:
self.sharedStrings = {}
elif s in self.sharedStrings:
return self.sharedStrings[s]
off = self.CreateString(s, encoding, errors)
self.sharedStrings[s] = off
return off
def CreateString(self, s, encoding="utf-8", errors="strict"):
"""CreateString writes a null-terminated byte string as a vector."""
self.assertNotNested()
## @cond FLATBUFFERS_INTERNAL
self.nested = True
## @endcond
if isinstance(s, compat.string_types):
x = s.encode(encoding, errors)
elif isinstance(s, compat.binary_types):
x = s
else:
raise TypeError("non-string passed to CreateString")
payload_len = len(x)
self.Prep(N.UOffsetTFlags.bytewidth, (payload_len + 1) * N.Uint8Flags.bytewidth)
self.Place(0, N.Uint8Flags)
new_head = self.head - payload_len
self.head = new_head
self.Bytes[new_head : new_head + payload_len] = x
self.vectorNumElems = payload_len
return self.EndVector()
def CreateByteVector(self, x):
"""CreateString writes a byte vector."""
self.assertNotNested()
## @cond FLATBUFFERS_INTERNAL
self.nested = True
## @endcond
if not isinstance(x, compat.binary_types):
raise TypeError("non-byte vector passed to CreateByteVector")
data_len = len(x)
self.Prep(N.UOffsetTFlags.bytewidth, data_len * N.Uint8Flags.bytewidth)
new_head = self.head - data_len
self.head = new_head
self.Bytes[new_head : new_head + data_len] = x
self.vectorNumElems = data_len
return self.EndVector()
def CreateNumpyVector(self, x):
"""CreateNumpyVector writes a numpy array into the buffer."""
if np is None:
# Numpy is required for this feature
raise NumpyRequiredForThisFeature("Numpy was not found.")
if not isinstance(x, np.ndarray):
raise TypeError("non-numpy-ndarray passed to CreateNumpyVector")
if x.dtype.kind not in ["b", "i", "u", "f"]:
raise TypeError("numpy-ndarray holds elements of unsupported datatype")
if x.ndim > 1:
raise TypeError("multidimensional-ndarray passed to CreateNumpyVector")
self.StartVector(x.itemsize, x.size, x.dtype.alignment)
# Ensure little endian byte ordering
if x.dtype.str[0] == "<":
x_lend = x
else:
x_lend = x.byteswap(inplace=False)
# tobytes ensures c_contiguous ordering
payload = x_lend.tobytes(order="C")
# Calculate total length
payload_len = len(payload)
new_head = self.head - payload_len
self.head = new_head
self.Bytes[new_head : new_head + payload_len] = payload
self.vectorNumElems = x.size
return self.EndVector()
## @cond FLATBUFFERS_INTERNAL
def assertNested(self):
"""Check that we are in the process of building an object."""
if not self.nested:
raise IsNotNestedError()
def assertNotNested(self):
"""Check that no other objects are being built while making this object.
If not, raise an exception.
"""
if self.nested:
raise IsNestedError()
def assertStructIsInline(self, obj):
"""Structs are always stored inline, so need to be created right
where they are used. You'll get this error if you created it
elsewhere.
"""
N.enforce_number(obj, N.UOffsetTFlags)
if obj != self.Offset():
msg = (
"flatbuffers: Tried to write a Struct at an Offset that "
"is different from the current Offset of the Builder."
)
raise StructIsNotInlineError(msg)
def Slot(self, slotnum):
"""Slot sets the vtable key `voffset` to the current location in the
buffer.
"""
self.assertNested()
self.current_vtable[slotnum] = self.Offset()
## @endcond
def __Finish(self, rootTable, sizePrefix, file_identifier=None):
"""Finish finalizes a buffer, pointing to the given `rootTable`."""
N.enforce_number(rootTable, N.UOffsetTFlags)
prepSize = N.UOffsetTFlags.bytewidth
if file_identifier is not None:
prepSize += N.Int32Flags.bytewidth
if sizePrefix:
prepSize += N.Int32Flags.bytewidth
self.Prep(self.minalign, prepSize)
if file_identifier is not None:
self.Prep(N.UOffsetTFlags.bytewidth, encode.FILE_IDENTIFIER_LENGTH)
# Convert bytes object file_identifier to an array of 4 8-bit integers,
# and use big-endian to enforce size compliance.
# https://docs.python.org/2/library/struct.html#format-characters
file_identifier = N.struct.unpack(">BBBB", file_identifier)
for i in range(encode.FILE_IDENTIFIER_LENGTH - 1, -1, -1):
# Place the bytes of the file_identifer in reverse order:
self.Place(file_identifier[i], N.Uint8Flags)
self.PrependUOffsetTRelative(rootTable)
if sizePrefix:
size = len(self.Bytes) - self.head
N.enforce_number(size, N.Int32Flags)
self.PrependInt32(size)
self.finished = True
return self.head
def Finish(self, rootTable, file_identifier=None):
"""Finish finalizes a buffer, pointing to the given `rootTable`."""
return self.__Finish(rootTable, False, file_identifier=file_identifier)
def FinishSizePrefixed(self, rootTable, file_identifier=None):
"""Finish finalizes a buffer, pointing to the given `rootTable`,
with the size prefixed.
"""
return self.__Finish(rootTable, True, file_identifier=file_identifier)
## @cond FLATBUFFERS_INTERNAL
def Prepend(self, flags, off):
self.Prep(flags.bytewidth, 0)
self.Place(off, flags)
def PrependSlot(self, flags, o, x, d):
if x is not None:
N.enforce_number(x, flags)
if d is not None:
N.enforce_number(d, flags)
if x != d or (self.forceDefaults and d is not None):
self.Prepend(flags, x)
self.Slot(o)
def PrependBoolSlot(self, *args):
self.PrependSlot(N.BoolFlags, *args)
def PrependByteSlot(self, *args):
self.PrependSlot(N.Uint8Flags, *args)
def PrependUint8Slot(self, *args):
self.PrependSlot(N.Uint8Flags, *args)
def PrependUint16Slot(self, *args):
self.PrependSlot(N.Uint16Flags, *args)
def PrependUint32Slot(self, *args):
self.PrependSlot(N.Uint32Flags, *args)
def PrependUint64Slot(self, *args):
self.PrependSlot(N.Uint64Flags, *args)
def PrependInt8Slot(self, *args):
self.PrependSlot(N.Int8Flags, *args)
def PrependInt16Slot(self, *args):
self.PrependSlot(N.Int16Flags, *args)
def PrependInt32Slot(self, *args):
self.PrependSlot(N.Int32Flags, *args)
def PrependInt64Slot(self, *args):
self.PrependSlot(N.Int64Flags, *args)
def PrependFloat32Slot(self, *args):
self.PrependSlot(N.Float32Flags, *args)
def PrependFloat64Slot(self, *args):
self.PrependSlot(N.Float64Flags, *args)
def PrependUOffsetTRelativeSlot(self, o, x, d):
"""PrependUOffsetTRelativeSlot prepends an UOffsetT onto the object at
vtable slot `o`. If value `x` equals default `d`, then the slot will
be set to zero and no other data will be written.
"""
if x != d or self.forceDefaults:
self.PrependUOffsetTRelative(x)
self.Slot(o)
def PrependStructSlot(self, v, x, d):
"""PrependStructSlot prepends a struct onto the object at vtable slot `o`.
Structs are stored inline, so nothing additional is being added. In
generated code, `d` is always 0.
"""
N.enforce_number(d, N.UOffsetTFlags)
if x != d:
self.assertStructIsInline(x)
self.Slot(v)
## @endcond
def PrependBool(self, x):
"""Prepend a `bool` to the Builder buffer.
Note: aligns and checks for space.
"""
self.Prepend(N.BoolFlags, x)
def PrependByte(self, x):
"""Prepend a `byte` to the Builder buffer.
Note: aligns and checks for space.
"""
self.Prepend(N.Uint8Flags, x)
def PrependUint8(self, x):
"""Prepend an `uint8` to the Builder buffer.
Note: aligns and checks for space.
"""
self.Prepend(N.Uint8Flags, x)
def PrependUint16(self, x):
"""Prepend an `uint16` to the Builder buffer.
Note: aligns and checks for space.
"""
self.Prepend(N.Uint16Flags, x)
def PrependUint32(self, x):
"""Prepend an `uint32` to the Builder buffer.
Note: aligns and checks for space.
"""
self.Prepend(N.Uint32Flags, x)
def PrependUint64(self, x):
"""Prepend an `uint64` to the Builder buffer.
Note: aligns and checks for space.
"""
self.Prepend(N.Uint64Flags, x)
def PrependInt8(self, x):
"""Prepend an `int8` to the Builder buffer.
Note: aligns and checks for space.
"""
self.Prepend(N.Int8Flags, x)
def PrependInt16(self, x):
"""Prepend an `int16` to the Builder buffer.
Note: aligns and checks for space.
"""
self.Prepend(N.Int16Flags, x)
def PrependInt32(self, x):
"""Prepend an `int32` to the Builder buffer.
Note: aligns and checks for space.
"""
self.Prepend(N.Int32Flags, x)
def PrependInt64(self, x):
"""Prepend an `int64` to the Builder buffer.
Note: aligns and checks for space.
"""
self.Prepend(N.Int64Flags, x)
def PrependFloat32(self, x):
"""Prepend a `float32` to the Builder buffer.
Note: aligns and checks for space.
"""
self.Prepend(N.Float32Flags, x)
def PrependFloat64(self, x):
"""Prepend a `float64` to the Builder buffer.
Note: aligns and checks for space.
"""
self.Prepend(N.Float64Flags, x)
def ForceDefaults(self, forceDefaults):
"""In order to save space, fields that are set to their default value
don't get serialized into the buffer. Forcing defaults provides a
way to manually disable this optimization. When set to `True`, will
always serialize default values.
"""
self.forceDefaults = forceDefaults
##############################################################
## @cond FLATBUFFERS_INTERNAL
def PrependVOffsetT(self, x):
self.Prepend(N.VOffsetTFlags, x)
def Place(self, x, flags):
"""Place prepends a value specified by `flags` to the Builder,
without checking for available space.
"""
N.enforce_number(x, flags)
new_head = self.head - flags.bytewidth
self.head = new_head
encode.Write(flags.packer_type, self.Bytes, new_head, x)
def PlaceVOffsetT(self, x):
"""PlaceVOffsetT prepends a VOffsetT to the Builder, without checking
for space.
"""
N.enforce_number(x, N.VOffsetTFlags)
new_head = self.head - N.VOffsetTFlags.bytewidth
self.head = new_head
encode.Write(packer.voffset, self.Bytes, new_head, x)
def PlaceSOffsetT(self, x):
"""PlaceSOffsetT prepends a SOffsetT to the Builder, without checking
for space.
"""
N.enforce_number(x, N.SOffsetTFlags)
new_head = self.head - N.SOffsetTFlags.bytewidth
self.head = new_head
encode.Write(packer.soffset, self.Bytes, new_head, x)
def PlaceUOffsetT(self, x):
"""PlaceUOffsetT prepends a UOffsetT to the Builder, without checking
for space.
"""
N.enforce_number(x, N.UOffsetTFlags)
new_head = self.head - N.UOffsetTFlags.bytewidth
self.head = new_head
encode.Write(packer.uoffset, self.Bytes, new_head, x)
## @endcond
## @}
@@ -0,0 +1,91 @@
# Copyright 2016 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""A tiny version of `six` to help with backwards compability.
Also includes compatibility helpers for numpy.
"""
import sys
PY2 = sys.version_info[0] == 2
PY26 = sys.version_info[0:2] == (2, 6)
PY27 = sys.version_info[0:2] == (2, 7)
PY275 = sys.version_info[0:3] >= (2, 7, 5)
PY3 = sys.version_info[0] == 3
PY34 = sys.version_info[0:2] >= (3, 4)
if PY3:
import importlib.machinery
string_types = (str,)
binary_types = (bytes, bytearray)
range_func = range
memoryview_type = memoryview
struct_bool_decl = "?"
else:
import imp
string_types = (unicode,)
if PY26 or PY27:
binary_types = (str, bytearray)
else:
binary_types = (str,)
range_func = xrange
if PY26 or (PY27 and not PY275):
memoryview_type = buffer
struct_bool_decl = "<b"
else:
memoryview_type = memoryview
struct_bool_decl = "?"
# Helper functions to facilitate making numpy optional instead of required
def import_numpy():
"""Returns the numpy module if it exists on the system,
otherwise returns None.
"""
if PY3:
numpy_exists = importlib.machinery.PathFinder.find_spec("numpy") is not None
else:
try:
imp.find_module("numpy")
numpy_exists = True
except ImportError:
numpy_exists = False
if numpy_exists:
# We do this outside of try/except block in case numpy exists
# but is not installed correctly. We do not want to catch an
# incorrect installation which would manifest as an
# ImportError.
import numpy as np
else:
np = None
return np
class NumpyRequiredForThisFeature(RuntimeError):
"""Error raised when user tries to use a feature that
requires numpy without having numpy installed.
"""
pass
# NOTE: Future Jython support may require code here (look at `six`).
@@ -0,0 +1,45 @@
# Copyright 2014 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from . import number_types as N
from . import packer
from .compat import memoryview_type
from .compat import NumpyRequiredForThisFeature, import_numpy
np = import_numpy()
FILE_IDENTIFIER_LENGTH = 4
def Get(packer_type, buf, head):
"""Get decodes a value at buf[head] using `packer_type`."""
return packer_type.unpack_from(memoryview_type(buf), head)[0]
def GetVectorAsNumpy(numpy_type, buf, count, offset):
"""GetVecAsNumpy decodes values starting at buf[head] as
`numpy_type`, where `numpy_type` is a numpy dtype.
"""
if np is not None:
# TODO: could set .flags.writeable = False to make users jump through
# hoops before modifying...
return np.frombuffer(buf, dtype=numpy_type, count=count, offset=offset)
else:
raise NumpyRequiredForThisFeature('Numpy was not found.')
def Write(packer_type, buf, head, n):
"""Write encodes `n` at buf[head] using `packer_type`."""
packer_type.pack_into(buf, head, n)
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,182 @@
# Copyright 2014 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import collections
import struct
from . import packer
from .compat import NumpyRequiredForThisFeature, import_numpy
np = import_numpy()
# For reference, see:
# https://docs.python.org/2/library/ctypes.html#ctypes-fundamental-data-types-2
# These classes could be collections.namedtuple instances, but those are new
# in 2.6 and we want to work towards 2.5 compatability.
class BoolFlags(object):
bytewidth = 1
min_val = False
max_val = True
py_type = bool
name = "bool"
packer_type = packer.boolean
class Uint8Flags(object):
bytewidth = 1
min_val = 0
max_val = (2**8) - 1
py_type = int
name = "uint8"
packer_type = packer.uint8
class Uint16Flags(object):
bytewidth = 2
min_val = 0
max_val = (2**16) - 1
py_type = int
name = "uint16"
packer_type = packer.uint16
class Uint32Flags(object):
bytewidth = 4
min_val = 0
max_val = (2**32) - 1
py_type = int
name = "uint32"
packer_type = packer.uint32
class Uint64Flags(object):
bytewidth = 8
min_val = 0
max_val = (2**64) - 1
py_type = int
name = "uint64"
packer_type = packer.uint64
class Int8Flags(object):
bytewidth = 1
min_val = -(2**7)
max_val = (2**7) - 1
py_type = int
name = "int8"
packer_type = packer.int8
class Int16Flags(object):
bytewidth = 2
min_val = -(2**15)
max_val = (2**15) - 1
py_type = int
name = "int16"
packer_type = packer.int16
class Int32Flags(object):
bytewidth = 4
min_val = -(2**31)
max_val = (2**31) - 1
py_type = int
name = "int32"
packer_type = packer.int32
class Int64Flags(object):
bytewidth = 8
min_val = -(2**63)
max_val = (2**63) - 1
py_type = int
name = "int64"
packer_type = packer.int64
class Float32Flags(object):
bytewidth = 4
min_val = None
max_val = None
py_type = float
name = "float32"
packer_type = packer.float32
class Float64Flags(object):
bytewidth = 8
min_val = None
max_val = None
py_type = float
name = "float64"
packer_type = packer.float64
class SOffsetTFlags(Int32Flags):
pass
class UOffsetTFlags(Uint32Flags):
pass
class VOffsetTFlags(Uint16Flags):
pass
def valid_number(n, flags):
if flags.min_val is None and flags.max_val is None:
return True
return flags.min_val <= n <= flags.max_val
def enforce_number(n, flags):
if flags.min_val is None and flags.max_val is None:
return
if not flags.min_val <= n <= flags.max_val:
raise TypeError("bad number %s for type %s" % (str(n), flags.name))
def float32_to_uint32(n):
packed = struct.pack("<1f", n)
(converted,) = struct.unpack("<1L", packed)
return converted
def uint32_to_float32(n):
packed = struct.pack("<1L", n)
(unpacked,) = struct.unpack("<1f", packed)
return unpacked
def float64_to_uint64(n):
packed = struct.pack("<1d", n)
(converted,) = struct.unpack("<1Q", packed)
return converted
def uint64_to_float64(n):
packed = struct.pack("<1Q", n)
(unpacked,) = struct.unpack("<1d", packed)
return unpacked
def to_numpy_type(number_type):
if np is not None:
return np.dtype(number_type.name).newbyteorder("<")
else:
raise NumpyRequiredForThisFeature("Numpy was not found.")
@@ -0,0 +1,41 @@
# Copyright 2016 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Provide pre-compiled struct packers for encoding and decoding.
See: https://docs.python.org/2/library/struct.html#format-characters
"""
import struct
from . import compat
boolean = struct.Struct(compat.struct_bool_decl)
uint8 = struct.Struct("<B")
uint16 = struct.Struct("<H")
uint32 = struct.Struct("<I")
uint64 = struct.Struct("<Q")
int8 = struct.Struct("<b")
int16 = struct.Struct("<h")
int32 = struct.Struct("<i")
int64 = struct.Struct("<q")
float32 = struct.Struct("<f")
float64 = struct.Struct("<d")
uoffset = uint32
soffset = int32
voffset = uint16
@@ -0,0 +1,148 @@
# Copyright 2014 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from . import encode
from . import number_types as N
class Table(object):
"""Table wraps a byte slice and provides read access to its data.
The variable `Pos` indicates the root of the FlatBuffers object therein.
"""
__slots__ = ("Bytes", "Pos")
def __init__(self, buf, pos):
N.enforce_number(pos, N.UOffsetTFlags)
self.Bytes = buf
self.Pos = pos
def Offset(self, vtableOffset):
"""Offset provides access into the Table's vtable.
Deprecated fields are ignored by checking the vtable's length.
"""
vtable = self.Pos - self.Get(N.SOffsetTFlags, self.Pos)
vtableEnd = self.Get(N.VOffsetTFlags, vtable)
if vtableOffset < vtableEnd:
return self.Get(N.VOffsetTFlags, vtable + vtableOffset)
return 0
def Indirect(self, off):
"""Indirect retrieves the relative offset stored at `offset`."""
N.enforce_number(off, N.UOffsetTFlags)
return off + encode.Get(N.UOffsetTFlags.packer_type, self.Bytes, off)
def String(self, off):
"""String gets a string from data stored inside the flatbuffer."""
N.enforce_number(off, N.UOffsetTFlags)
off += encode.Get(N.UOffsetTFlags.packer_type, self.Bytes, off)
start = off + N.UOffsetTFlags.bytewidth
length = encode.Get(N.UOffsetTFlags.packer_type, self.Bytes, off)
return bytes(self.Bytes[start : start + length])
def VectorLen(self, off):
"""VectorLen retrieves the length of the vector whose offset is stored
at "off" in this object.
"""
N.enforce_number(off, N.UOffsetTFlags)
off += self.Pos
off += encode.Get(N.UOffsetTFlags.packer_type, self.Bytes, off)
ret = encode.Get(N.UOffsetTFlags.packer_type, self.Bytes, off)
return ret
def Vector(self, off):
"""Vector retrieves the start of data of the vector whose offset is
stored at "off" in this object.
"""
N.enforce_number(off, N.UOffsetTFlags)
off += self.Pos
x = off + self.Get(N.UOffsetTFlags, off)
# data starts after metadata containing the vector length
x += N.UOffsetTFlags.bytewidth
return x
def Union(self, t2, off):
"""Union initializes any Table-derived type to point to the union at
the given offset.
"""
assert type(t2) is Table
N.enforce_number(off, N.UOffsetTFlags)
off += self.Pos
t2.Pos = off + self.Get(N.UOffsetTFlags, off)
t2.Bytes = self.Bytes
def Get(self, flags, off):
"""Get retrieves a value of the type specified by `flags` at the
given offset.
"""
N.enforce_number(off, N.UOffsetTFlags)
return flags.py_type(encode.Get(flags.packer_type, self.Bytes, off))
def GetSlot(self, slot, d, validator_flags):
N.enforce_number(slot, N.VOffsetTFlags)
if validator_flags is not None:
N.enforce_number(d, validator_flags)
off = self.Offset(slot)
if off == 0:
return d
return self.Get(validator_flags, self.Pos + off)
def GetVectorAsNumpy(self, flags, off):
"""GetVectorAsNumpy returns the vector that starts at `Vector(off)`
as a numpy array with the type specified by `flags`. The array is
a `view` into Bytes, so modifying the returned array will
modify Bytes in place.
"""
offset = self.Vector(off)
length = self.VectorLen(off) # TODO: length accounts for bytewidth, right?
numpy_dtype = N.to_numpy_type(flags)
return encode.GetVectorAsNumpy(numpy_dtype, self.Bytes, length, offset)
def GetArrayAsNumpy(self, flags, off, length):
"""GetArrayAsNumpy returns the array with fixed width that starts at `Vector(offset)`
with length `length` as a numpy array with the type specified by `flags`.
The
array is a `view` into Bytes so modifying the returned will modify Bytes in
place.
"""
numpy_dtype = N.to_numpy_type(flags)
return encode.GetVectorAsNumpy(numpy_dtype, self.Bytes, length, off)
def GetVOffsetTSlot(self, slot, d):
"""GetVOffsetTSlot retrieves the VOffsetT that the given vtable location
points to. If the vtable value is zero, the default value `d`
will be returned.
"""
N.enforce_number(slot, N.VOffsetTFlags)
N.enforce_number(d, N.VOffsetTFlags)
off = self.Offset(slot)
if off == 0:
return d
return off
@@ -0,0 +1,47 @@
# Copyright 2017 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from . import encode
from . import number_types
from . import packer
def GetSizePrefix(buf, offset):
"""Extract the size prefix from a buffer."""
return encode.Get(packer.int32, buf, offset)
def GetBufferIdentifier(buf, offset, size_prefixed=False):
"""Extract the file_identifier from a buffer"""
if size_prefixed:
# increase offset by size of UOffsetTFlags
offset += number_types.UOffsetTFlags.bytewidth
# increase offset by size of root table pointer
offset += number_types.UOffsetTFlags.bytewidth
# end of FILE_IDENTIFIER
end = offset + encode.FILE_IDENTIFIER_LENGTH
return buf[offset:end]
def BufferHasIdentifier(buf, offset, file_identifier, size_prefixed=False):
got = GetBufferIdentifier(buf, offset, size_prefixed=size_prefixed)
return got == file_identifier
def RemoveSizePrefix(buf, offset):
"""Create a slice of a size-prefixed buffer that has
its position advanced just past the size prefix.
"""
return buf, offset + number_types.Int32Flags.bytewidth
Binary file not shown.
@@ -0,0 +1,10 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
# Copyright 2007 Google Inc. All Rights Reserved.
__version__ = '6.33.5'
@@ -0,0 +1,53 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Contains the Any helper APIs."""
from typing import Optional, TypeVar
from google.protobuf import descriptor
from google.protobuf.message import Message
from google.protobuf.any_pb2 import Any
_MessageT = TypeVar('_MessageT', bound=Message)
def pack(
msg: Message,
type_url_prefix: Optional[str] = 'type.googleapis.com/',
deterministic: Optional[bool] = None,
) -> Any:
any_msg = Any()
any_msg.Pack(
msg=msg, type_url_prefix=type_url_prefix, deterministic=deterministic
)
return any_msg
def unpack(any_msg: Any, msg: Message) -> bool:
return any_msg.Unpack(msg=msg)
def unpack_as(any_msg: Any, message_type: type[_MessageT]) -> _MessageT:
unpacked = message_type()
if unpack(any_msg, unpacked):
return unpacked
else:
raise TypeError(
f'Attempted to unpack {type_name(any_msg)} to'
f' {message_type.__qualname__}'
)
def type_name(any_msg: Any) -> str:
return any_msg.TypeName()
def is_type(any_msg: Any, des: descriptor.Descriptor) -> bool:
return any_msg.Is(des)
@@ -0,0 +1,37 @@
# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# NO CHECKED-IN PROTOBUF GENCODE
# source: google/protobuf/any.proto
# Protobuf Python Version: 6.33.5
"""Generated protocol buffer code."""
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import runtime_version as _runtime_version
from google.protobuf import symbol_database as _symbol_database
from google.protobuf.internal import builder as _builder
_runtime_version.ValidateProtobufRuntimeVersion(
_runtime_version.Domain.PUBLIC,
6,
33,
5,
'',
'google/protobuf/any.proto'
)
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x19google/protobuf/any.proto\x12\x0fgoogle.protobuf\"6\n\x03\x41ny\x12\x19\n\x08type_url\x18\x01 \x01(\tR\x07typeUrl\x12\x14\n\x05value\x18\x02 \x01(\x0cR\x05valueBv\n\x13\x63om.google.protobufB\x08\x41nyProtoP\x01Z,google.golang.org/protobuf/types/known/anypb\xa2\x02\x03GPB\xaa\x02\x1eGoogle.Protobuf.WellKnownTypesb\x06proto3')
_globals = globals()
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'google.protobuf.any_pb2', _globals)
if not _descriptor._USE_C_DESCRIPTORS:
_globals['DESCRIPTOR']._loaded_options = None
_globals['DESCRIPTOR']._serialized_options = b'\n\023com.google.protobufB\010AnyProtoP\001Z,google.golang.org/protobuf/types/known/anypb\242\002\003GPB\252\002\036Google.Protobuf.WellKnownTypes'
_globals['_ANY']._serialized_start=46
_globals['_ANY']._serialized_end=100
# @@protoc_insertion_point(module_scope)
@@ -0,0 +1,47 @@
# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# NO CHECKED-IN PROTOBUF GENCODE
# source: google/protobuf/api.proto
# Protobuf Python Version: 6.33.5
"""Generated protocol buffer code."""
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import runtime_version as _runtime_version
from google.protobuf import symbol_database as _symbol_database
from google.protobuf.internal import builder as _builder
_runtime_version.ValidateProtobufRuntimeVersion(
_runtime_version.Domain.PUBLIC,
6,
33,
5,
'',
'google/protobuf/api.proto'
)
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
from google.protobuf import source_context_pb2 as google_dot_protobuf_dot_source__context__pb2
from google.protobuf import type_pb2 as google_dot_protobuf_dot_type__pb2
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x19google/protobuf/api.proto\x12\x0fgoogle.protobuf\x1a$google/protobuf/source_context.proto\x1a\x1agoogle/protobuf/type.proto\"\xdb\x02\n\x03\x41pi\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\x12\x31\n\x07methods\x18\x02 \x03(\x0b\x32\x17.google.protobuf.MethodR\x07methods\x12\x31\n\x07options\x18\x03 \x03(\x0b\x32\x17.google.protobuf.OptionR\x07options\x12\x18\n\x07version\x18\x04 \x01(\tR\x07version\x12\x45\n\x0esource_context\x18\x05 \x01(\x0b\x32\x1e.google.protobuf.SourceContextR\rsourceContext\x12.\n\x06mixins\x18\x06 \x03(\x0b\x32\x16.google.protobuf.MixinR\x06mixins\x12/\n\x06syntax\x18\x07 \x01(\x0e\x32\x17.google.protobuf.SyntaxR\x06syntax\x12\x18\n\x07\x65\x64ition\x18\x08 \x01(\tR\x07\x65\x64ition\"\xd4\x02\n\x06Method\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\x12(\n\x10request_type_url\x18\x02 \x01(\tR\x0erequestTypeUrl\x12+\n\x11request_streaming\x18\x03 \x01(\x08R\x10requestStreaming\x12*\n\x11response_type_url\x18\x04 \x01(\tR\x0fresponseTypeUrl\x12-\n\x12response_streaming\x18\x05 \x01(\x08R\x11responseStreaming\x12\x31\n\x07options\x18\x06 \x03(\x0b\x32\x17.google.protobuf.OptionR\x07options\x12\x33\n\x06syntax\x18\x07 \x01(\x0e\x32\x17.google.protobuf.SyntaxB\x02\x18\x01R\x06syntax\x12\x1c\n\x07\x65\x64ition\x18\x08 \x01(\tB\x02\x18\x01R\x07\x65\x64ition\"/\n\x05Mixin\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\x12\x12\n\x04root\x18\x02 \x01(\tR\x04rootBv\n\x13\x63om.google.protobufB\x08\x41piProtoP\x01Z,google.golang.org/protobuf/types/known/apipb\xa2\x02\x03GPB\xaa\x02\x1eGoogle.Protobuf.WellKnownTypesb\x06proto3')
_globals = globals()
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'google.protobuf.api_pb2', _globals)
if not _descriptor._USE_C_DESCRIPTORS:
_globals['DESCRIPTOR']._loaded_options = None
_globals['DESCRIPTOR']._serialized_options = b'\n\023com.google.protobufB\010ApiProtoP\001Z,google.golang.org/protobuf/types/known/apipb\242\002\003GPB\252\002\036Google.Protobuf.WellKnownTypes'
_globals['_METHOD'].fields_by_name['syntax']._loaded_options = None
_globals['_METHOD'].fields_by_name['syntax']._serialized_options = b'\030\001'
_globals['_METHOD'].fields_by_name['edition']._loaded_options = None
_globals['_METHOD'].fields_by_name['edition']._serialized_options = b'\030\001'
_globals['_API']._serialized_start=113
_globals['_API']._serialized_end=460
_globals['_METHOD']._serialized_start=463
_globals['_METHOD']._serialized_end=803
_globals['_MIXIN']._serialized_start=805
_globals['_MIXIN']._serialized_end=852
# @@protoc_insertion_point(module_scope)
@@ -0,0 +1,46 @@
# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# NO CHECKED-IN PROTOBUF GENCODE
# source: google/protobuf/compiler/plugin.proto
# Protobuf Python Version: 6.33.5
"""Generated protocol buffer code."""
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import runtime_version as _runtime_version
from google.protobuf import symbol_database as _symbol_database
from google.protobuf.internal import builder as _builder
_runtime_version.ValidateProtobufRuntimeVersion(
_runtime_version.Domain.PUBLIC,
6,
33,
5,
'',
'google/protobuf/compiler/plugin.proto'
)
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
from google.protobuf import descriptor_pb2 as google_dot_protobuf_dot_descriptor__pb2
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n%google/protobuf/compiler/plugin.proto\x12\x18google.protobuf.compiler\x1a google/protobuf/descriptor.proto\"c\n\x07Version\x12\x14\n\x05major\x18\x01 \x01(\x05R\x05major\x12\x14\n\x05minor\x18\x02 \x01(\x05R\x05minor\x12\x14\n\x05patch\x18\x03 \x01(\x05R\x05patch\x12\x16\n\x06suffix\x18\x04 \x01(\tR\x06suffix\"\xcf\x02\n\x14\x43odeGeneratorRequest\x12(\n\x10\x66ile_to_generate\x18\x01 \x03(\tR\x0e\x66ileToGenerate\x12\x1c\n\tparameter\x18\x02 \x01(\tR\tparameter\x12\x43\n\nproto_file\x18\x0f \x03(\x0b\x32$.google.protobuf.FileDescriptorProtoR\tprotoFile\x12\\\n\x17source_file_descriptors\x18\x11 \x03(\x0b\x32$.google.protobuf.FileDescriptorProtoR\x15sourceFileDescriptors\x12L\n\x10\x63ompiler_version\x18\x03 \x01(\x0b\x32!.google.protobuf.compiler.VersionR\x0f\x63ompilerVersion\"\x85\x04\n\x15\x43odeGeneratorResponse\x12\x14\n\x05\x65rror\x18\x01 \x01(\tR\x05\x65rror\x12-\n\x12supported_features\x18\x02 \x01(\x04R\x11supportedFeatures\x12\'\n\x0fminimum_edition\x18\x03 \x01(\x05R\x0eminimumEdition\x12\'\n\x0fmaximum_edition\x18\x04 \x01(\x05R\x0emaximumEdition\x12H\n\x04\x66ile\x18\x0f \x03(\x0b\x32\x34.google.protobuf.compiler.CodeGeneratorResponse.FileR\x04\x66ile\x1a\xb1\x01\n\x04\x46ile\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\x12\'\n\x0finsertion_point\x18\x02 \x01(\tR\x0einsertionPoint\x12\x18\n\x07\x63ontent\x18\x0f \x01(\tR\x07\x63ontent\x12R\n\x13generated_code_info\x18\x10 \x01(\x0b\x32\".google.protobuf.GeneratedCodeInfoR\x11generatedCodeInfo\"W\n\x07\x46\x65\x61ture\x12\x10\n\x0c\x46\x45\x41TURE_NONE\x10\x00\x12\x1b\n\x17\x46\x45\x41TURE_PROTO3_OPTIONAL\x10\x01\x12\x1d\n\x19\x46\x45\x41TURE_SUPPORTS_EDITIONS\x10\x02\x42r\n\x1c\x63om.google.protobuf.compilerB\x0cPluginProtosZ)google.golang.org/protobuf/types/pluginpb\xaa\x02\x18Google.Protobuf.Compiler')
_globals = globals()
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'google.protobuf.compiler.plugin_pb2', _globals)
if not _descriptor._USE_C_DESCRIPTORS:
_globals['DESCRIPTOR']._loaded_options = None
_globals['DESCRIPTOR']._serialized_options = b'\n\034com.google.protobuf.compilerB\014PluginProtosZ)google.golang.org/protobuf/types/pluginpb\252\002\030Google.Protobuf.Compiler'
_globals['_VERSION']._serialized_start=101
_globals['_VERSION']._serialized_end=200
_globals['_CODEGENERATORREQUEST']._serialized_start=203
_globals['_CODEGENERATORREQUEST']._serialized_end=538
_globals['_CODEGENERATORRESPONSE']._serialized_start=541
_globals['_CODEGENERATORRESPONSE']._serialized_end=1058
_globals['_CODEGENERATORRESPONSE_FILE']._serialized_start=792
_globals['_CODEGENERATORRESPONSE_FILE']._serialized_end=969
_globals['_CODEGENERATORRESPONSE_FEATURE']._serialized_start=971
_globals['_CODEGENERATORRESPONSE_FEATURE']._serialized_end=1058
# @@protoc_insertion_point(module_scope)
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,172 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Provides a container for DescriptorProtos."""
__author__ = 'matthewtoia@google.com (Matt Toia)'
import warnings
class Error(Exception):
pass
class DescriptorDatabaseConflictingDefinitionError(Error):
"""Raised when a proto is added with the same name & different descriptor."""
class DescriptorDatabase(object):
"""A container accepting FileDescriptorProtos and maps DescriptorProtos."""
def __init__(self):
self._file_desc_protos_by_file = {}
self._file_desc_protos_by_symbol = {}
def Add(self, file_desc_proto):
"""Adds the FileDescriptorProto and its types to this database.
Args:
file_desc_proto: The FileDescriptorProto to add.
Raises:
DescriptorDatabaseConflictingDefinitionError: if an attempt is made to
add a proto with the same name but different definition than an
existing proto in the database.
"""
proto_name = file_desc_proto.name
if proto_name not in self._file_desc_protos_by_file:
self._file_desc_protos_by_file[proto_name] = file_desc_proto
elif self._file_desc_protos_by_file[proto_name] != file_desc_proto:
raise DescriptorDatabaseConflictingDefinitionError(
'%s already added, but with different descriptor.' % proto_name)
else:
return
# Add all the top-level descriptors to the index.
package = file_desc_proto.package
for message in file_desc_proto.message_type:
for name in _ExtractSymbols(message, package):
self._AddSymbol(name, file_desc_proto)
for enum in file_desc_proto.enum_type:
self._AddSymbol(
('.'.join((package, enum.name)) if package else enum.name),
file_desc_proto,
)
for enum_value in enum.value:
self._file_desc_protos_by_symbol[
'.'.join((package, enum_value.name)) if package else enum_value.name
] = file_desc_proto
for extension in file_desc_proto.extension:
self._AddSymbol(
('.'.join((package, extension.name)) if package else extension.name),
file_desc_proto,
)
for service in file_desc_proto.service:
self._AddSymbol(
('.'.join((package, service.name)) if package else service.name),
file_desc_proto,
)
def FindFileByName(self, name):
"""Finds the file descriptor proto by file name.
Typically the file name is a relative path ending to a .proto file. The
proto with the given name will have to have been added to this database
using the Add method or else an error will be raised.
Args:
name: The file name to find.
Returns:
The file descriptor proto matching the name.
Raises:
KeyError if no file by the given name was added.
"""
return self._file_desc_protos_by_file[name]
def FindFileContainingSymbol(self, symbol):
"""Finds the file descriptor proto containing the specified symbol.
The symbol should be a fully qualified name including the file descriptor's
package and any containing messages. Some examples:
'some.package.name.Message'
'some.package.name.Message.NestedEnum'
'some.package.name.Message.some_field'
The file descriptor proto containing the specified symbol must be added to
this database using the Add method or else an error will be raised.
Args:
symbol: The fully qualified symbol name.
Returns:
The file descriptor proto containing the symbol.
Raises:
KeyError if no file contains the specified symbol.
"""
if symbol.count('.') == 1 and symbol[0] == '.':
symbol = symbol.lstrip('.')
warnings.warn(
'Please remove the leading "." when '
'FindFileContainingSymbol, this will turn to error '
'in 2026 Jan.',
RuntimeWarning,
)
try:
return self._file_desc_protos_by_symbol[symbol]
except KeyError:
# Fields, enum values, and nested extensions are not in
# _file_desc_protos_by_symbol. Try to find the top level
# descriptor. Non-existent nested symbol under a valid top level
# descriptor can also be found. The behavior is the same with
# protobuf C++.
top_level, _, _ = symbol.rpartition('.')
try:
return self._file_desc_protos_by_symbol[top_level]
except KeyError:
# Raise the original symbol as a KeyError for better diagnostics.
raise KeyError(symbol)
def FindFileContainingExtension(self, extendee_name, extension_number):
# TODO: implement this API.
return None
def FindAllExtensionNumbers(self, extendee_name):
# TODO: implement this API.
return []
def _AddSymbol(self, name, file_desc_proto):
if name in self._file_desc_protos_by_symbol:
warn_msg = ('Conflict register for file "' + file_desc_proto.name +
'": ' + name +
' is already defined in file "' +
self._file_desc_protos_by_symbol[name].name + '"')
warnings.warn(warn_msg, RuntimeWarning)
self._file_desc_protos_by_symbol[name] = file_desc_proto
def _ExtractSymbols(desc_proto, package):
"""Pulls out all the symbols from a descriptor proto.
Args:
desc_proto: The proto to extract symbols from.
package: The package containing the descriptor type.
Yields:
The fully qualified name found in the descriptor.
"""
message_name = package + '.' + desc_proto.name if package else desc_proto.name
yield message_name
for nested_type in desc_proto.nested_type:
for symbol in _ExtractSymbols(nested_type, message_name):
yield symbol
for enum_type in desc_proto.enum_type:
yield '.'.join((message_name, enum_type.name))
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,100 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Contains the Duration helper APIs."""
import datetime
from google.protobuf.duration_pb2 import Duration
def from_json_string(value: str) -> Duration:
"""Converts a string to Duration.
Args:
value: A string to be converted. The string must end with 's'. Any
fractional digits (or none) are accepted as long as they fit into
precision. For example: "1s", "1.01s", "1.0000001s", "-3.100s"
Raises:
ValueError: On parsing problems.
"""
duration = Duration()
duration.FromJsonString(value)
return duration
def from_microseconds(micros: float) -> Duration:
"""Converts microseconds to Duration."""
duration = Duration()
duration.FromMicroseconds(micros)
return duration
def from_milliseconds(millis: float) -> Duration:
"""Converts milliseconds to Duration."""
duration = Duration()
duration.FromMilliseconds(millis)
return duration
def from_nanoseconds(nanos: float) -> Duration:
"""Converts nanoseconds to Duration."""
duration = Duration()
duration.FromNanoseconds(nanos)
return duration
def from_seconds(seconds: float) -> Duration:
"""Converts seconds to Duration."""
duration = Duration()
duration.FromSeconds(seconds)
return duration
def from_timedelta(td: datetime.timedelta) -> Duration:
"""Converts timedelta to Duration."""
duration = Duration()
duration.FromTimedelta(td)
return duration
def to_json_string(duration: Duration) -> str:
"""Converts Duration to string format.
Returns:
A string converted from self. The string format will contains
3, 6, or 9 fractional digits depending on the precision required to
represent the exact Duration value. For example: "1s", "1.010s",
"1.000000100s", "-3.100s"
"""
return duration.ToJsonString()
def to_microseconds(duration: Duration) -> int:
"""Converts a Duration to microseconds."""
return duration.ToMicroseconds()
def to_milliseconds(duration: Duration) -> int:
"""Converts a Duration to milliseconds."""
return duration.ToMilliseconds()
def to_nanoseconds(duration: Duration) -> int:
"""Converts a Duration to nanoseconds."""
return duration.ToNanoseconds()
def to_seconds(duration: Duration) -> int:
"""Converts a Duration to seconds."""
return duration.ToSeconds()
def to_timedelta(duration: Duration) -> datetime.timedelta:
"""Converts Duration to timedelta."""
return duration.ToTimedelta()
@@ -0,0 +1,37 @@
# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# NO CHECKED-IN PROTOBUF GENCODE
# source: google/protobuf/duration.proto
# Protobuf Python Version: 6.33.5
"""Generated protocol buffer code."""
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import runtime_version as _runtime_version
from google.protobuf import symbol_database as _symbol_database
from google.protobuf.internal import builder as _builder
_runtime_version.ValidateProtobufRuntimeVersion(
_runtime_version.Domain.PUBLIC,
6,
33,
5,
'',
'google/protobuf/duration.proto'
)
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1egoogle/protobuf/duration.proto\x12\x0fgoogle.protobuf\":\n\x08\x44uration\x12\x18\n\x07seconds\x18\x01 \x01(\x03R\x07seconds\x12\x14\n\x05nanos\x18\x02 \x01(\x05R\x05nanosB\x83\x01\n\x13\x63om.google.protobufB\rDurationProtoP\x01Z1google.golang.org/protobuf/types/known/durationpb\xf8\x01\x01\xa2\x02\x03GPB\xaa\x02\x1eGoogle.Protobuf.WellKnownTypesb\x06proto3')
_globals = globals()
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'google.protobuf.duration_pb2', _globals)
if not _descriptor._USE_C_DESCRIPTORS:
_globals['DESCRIPTOR']._loaded_options = None
_globals['DESCRIPTOR']._serialized_options = b'\n\023com.google.protobufB\rDurationProtoP\001Z1google.golang.org/protobuf/types/known/durationpb\370\001\001\242\002\003GPB\252\002\036Google.Protobuf.WellKnownTypes'
_globals['_DURATION']._serialized_start=51
_globals['_DURATION']._serialized_end=109
# @@protoc_insertion_point(module_scope)
@@ -0,0 +1,37 @@
# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# NO CHECKED-IN PROTOBUF GENCODE
# source: google/protobuf/empty.proto
# Protobuf Python Version: 6.33.5
"""Generated protocol buffer code."""
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import runtime_version as _runtime_version
from google.protobuf import symbol_database as _symbol_database
from google.protobuf.internal import builder as _builder
_runtime_version.ValidateProtobufRuntimeVersion(
_runtime_version.Domain.PUBLIC,
6,
33,
5,
'',
'google/protobuf/empty.proto'
)
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1bgoogle/protobuf/empty.proto\x12\x0fgoogle.protobuf\"\x07\n\x05\x45mptyB}\n\x13\x63om.google.protobufB\nEmptyProtoP\x01Z.google.golang.org/protobuf/types/known/emptypb\xf8\x01\x01\xa2\x02\x03GPB\xaa\x02\x1eGoogle.Protobuf.WellKnownTypesb\x06proto3')
_globals = globals()
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'google.protobuf.empty_pb2', _globals)
if not _descriptor._USE_C_DESCRIPTORS:
_globals['DESCRIPTOR']._loaded_options = None
_globals['DESCRIPTOR']._serialized_options = b'\n\023com.google.protobufB\nEmptyProtoP\001Z.google.golang.org/protobuf/types/known/emptypb\370\001\001\242\002\003GPB\252\002\036Google.Protobuf.WellKnownTypes'
_globals['_EMPTY']._serialized_start=48
_globals['_EMPTY']._serialized_end=55
# @@protoc_insertion_point(module_scope)
@@ -0,0 +1,37 @@
# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# NO CHECKED-IN PROTOBUF GENCODE
# source: google/protobuf/field_mask.proto
# Protobuf Python Version: 6.33.5
"""Generated protocol buffer code."""
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import runtime_version as _runtime_version
from google.protobuf import symbol_database as _symbol_database
from google.protobuf.internal import builder as _builder
_runtime_version.ValidateProtobufRuntimeVersion(
_runtime_version.Domain.PUBLIC,
6,
33,
5,
'',
'google/protobuf/field_mask.proto'
)
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n google/protobuf/field_mask.proto\x12\x0fgoogle.protobuf\"!\n\tFieldMask\x12\x14\n\x05paths\x18\x01 \x03(\tR\x05pathsB\x85\x01\n\x13\x63om.google.protobufB\x0e\x46ieldMaskProtoP\x01Z2google.golang.org/protobuf/types/known/fieldmaskpb\xf8\x01\x01\xa2\x02\x03GPB\xaa\x02\x1eGoogle.Protobuf.WellKnownTypesb\x06proto3')
_globals = globals()
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'google.protobuf.field_mask_pb2', _globals)
if not _descriptor._USE_C_DESCRIPTORS:
_globals['DESCRIPTOR']._loaded_options = None
_globals['DESCRIPTOR']._serialized_options = b'\n\023com.google.protobufB\016FieldMaskProtoP\001Z2google.golang.org/protobuf/types/known/fieldmaskpb\370\001\001\242\002\003GPB\252\002\036Google.Protobuf.WellKnownTypes'
_globals['_FIELDMASK']._serialized_start=53
_globals['_FIELDMASK']._serialized_end=86
# @@protoc_insertion_point(module_scope)
@@ -0,0 +1,7 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
@@ -0,0 +1,136 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Determine which implementation of the protobuf API is used in this process.
"""
import importlib
import os
import sys
import warnings
_GOOGLE3_PYTHON_UPB_DEFAULT = True
def _ApiVersionToImplementationType(api_version):
if api_version == 2:
return 'cpp'
if api_version == 1:
raise ValueError('api_version=1 is no longer supported.')
if api_version == 0:
return 'python'
return None
_implementation_type = None
try:
# pylint: disable=g-import-not-at-top
from google.protobuf.internal import _api_implementation
# The compile-time constants in the _api_implementation module can be used to
# switch to a certain implementation of the Python API at build time.
_implementation_type = _ApiVersionToImplementationType(
_api_implementation.api_version)
except ImportError:
pass # Unspecified by compiler flags.
def _CanImport(mod_name):
try:
mod = importlib.import_module(mod_name)
# Work around a known issue in the classic bootstrap .par import hook.
if not mod:
raise ImportError(mod_name + ' import succeeded but was None')
return True
except ImportError:
return False
if _implementation_type is None:
if _CanImport('google._upb._message'):
_implementation_type = 'upb'
elif _CanImport('google.protobuf.pyext._message'):
_implementation_type = 'cpp'
else:
_implementation_type = 'python'
# This environment variable can be used to switch to a certain implementation
# of the Python API, overriding the compile-time constants in the
# _api_implementation module. Right now only 'python', 'cpp' and 'upb' are
# valid values. Any other value will raise error.
_implementation_type = os.getenv('PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION',
_implementation_type)
if _implementation_type not in ('python', 'cpp', 'upb'):
raise ValueError('PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION {0} is not '
'supported. Please set to \'python\', \'cpp\' or '
'\'upb\'.'.format(_implementation_type))
if 'PyPy' in sys.version and _implementation_type == 'cpp':
warnings.warn('PyPy does not work yet with cpp protocol buffers. '
'Falling back to the python implementation.')
_implementation_type = 'python'
_c_module = None
if _implementation_type == 'cpp':
try:
# pylint: disable=g-import-not-at-top
from google.protobuf.pyext import _message
sys.modules['google3.net.proto2.python.internal.cpp._message'] = _message
_c_module = _message
del _message
except ImportError:
# TODO: fail back to python
warnings.warn(
'Selected implementation cpp is not available.')
pass
if _implementation_type == 'upb':
try:
# pylint: disable=g-import-not-at-top
from google._upb import _message
_c_module = _message
del _message
except ImportError:
warnings.warn('Selected implementation upb is not available. '
'Falling back to the python implementation.')
_implementation_type = 'python'
pass
# Detect if serialization should be deterministic by default
try:
# The presence of this module in a build allows the proto implementation to
# be upgraded merely via build deps.
#
# NOTE: Merely importing this automatically enables deterministic proto
# serialization for C++ code, but we still need to export it as a boolean so
# that we can do the same for `_implementation_type == 'python'`.
#
# NOTE2: It is possible for C++ code to enable deterministic serialization by
# default _without_ affecting Python code, if the C++ implementation is not in
# use by this module. That is intended behavior, so we don't actually expose
# this boolean outside of this module.
#
# pylint: disable=g-import-not-at-top,unused-import
from google.protobuf import enable_deterministic_proto_serialization
_python_deterministic_proto_serialization = True
except ImportError:
_python_deterministic_proto_serialization = False
# Usage of this function is discouraged. Clients shouldn't care which
# implementation of the API is in use. Note that there is no guarantee
# that differences between APIs will be maintained.
# Please don't use this function if possible.
def Type():
return _implementation_type
# For internal use only
def IsPythonDefaultSerializationDeterministic():
return _python_deterministic_proto_serialization
@@ -0,0 +1,118 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Builds descriptors, message classes and services for generated _pb2.py.
This file is only called in python generated _pb2.py files. It builds
descriptors, message classes and services that users can directly use
in generated code.
"""
__author__ = 'jieluo@google.com (Jie Luo)'
from google.protobuf.internal import enum_type_wrapper
from google.protobuf.internal import python_message
from google.protobuf import message as _message
from google.protobuf import reflection as _reflection
from google.protobuf import symbol_database as _symbol_database
_sym_db = _symbol_database.Default()
def BuildMessageAndEnumDescriptors(file_des, module):
"""Builds message and enum descriptors.
Args:
file_des: FileDescriptor of the .proto file
module: Generated _pb2 module
"""
def BuildNestedDescriptors(msg_des, prefix):
for (name, nested_msg) in msg_des.nested_types_by_name.items():
module_name = prefix + name.upper()
module[module_name] = nested_msg
BuildNestedDescriptors(nested_msg, module_name + '_')
for enum_des in msg_des.enum_types:
module[prefix + enum_des.name.upper()] = enum_des
for (name, msg_des) in file_des.message_types_by_name.items():
module_name = '_' + name.upper()
module[module_name] = msg_des
BuildNestedDescriptors(msg_des, module_name + '_')
def BuildTopDescriptorsAndMessages(file_des, module_name, module):
"""Builds top level descriptors and message classes.
Args:
file_des: FileDescriptor of the .proto file
module_name: str, the name of generated _pb2 module
module: Generated _pb2 module
"""
def BuildMessage(msg_des, prefix):
create_dict = {}
for (name, nested_msg) in msg_des.nested_types_by_name.items():
create_dict[name] = BuildMessage(nested_msg, prefix + msg_des.name + '.')
create_dict['DESCRIPTOR'] = msg_des
create_dict['__module__'] = module_name
create_dict['__qualname__'] = prefix + msg_des.name
message_class = _reflection.GeneratedProtocolMessageType(
msg_des.name, (_message.Message,), create_dict)
_sym_db.RegisterMessage(message_class)
return message_class
# top level enums
for (name, enum_des) in file_des.enum_types_by_name.items():
module['_' + name.upper()] = enum_des
module[name] = enum_type_wrapper.EnumTypeWrapper(enum_des)
for enum_value in enum_des.values:
module[enum_value.name] = enum_value.number
# top level extensions
for (name, extension_des) in file_des.extensions_by_name.items():
module[name.upper() + '_FIELD_NUMBER'] = extension_des.number
module[name] = extension_des
# services
for (name, service) in file_des.services_by_name.items():
module['_' + name.upper()] = service
# Build messages.
for (name, msg_des) in file_des.message_types_by_name.items():
module[name] = BuildMessage(msg_des, '')
def AddHelpersToExtensions(file_des):
"""no-op to keep old generated code work with new runtime.
Args:
file_des: FileDescriptor of the .proto file
"""
# TODO: Remove this on-op
return
def BuildServices(file_des, module_name, module):
"""Builds services classes and services stub class.
Args:
file_des: FileDescriptor of the .proto file
module_name: str, the name of generated _pb2 module
module: Generated _pb2 module
"""
# pylint: disable=g-import-not-at-top
from google.protobuf import service_reflection
# pylint: enable=g-import-not-at-top
for (name, service) in file_des.services_by_name.items():
module[name] = service_reflection.GeneratedServiceType(
name, (),
dict(DESCRIPTOR=service, __module__=module_name))
stub_name = name + '_Stub'
module[stub_name] = service_reflection.GeneratedServiceStubType(
stub_name, (module[name],),
dict(DESCRIPTOR=service, __module__=module_name))
@@ -0,0 +1,690 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Contains container classes to represent different protocol buffer types.
This file defines container classes which represent categories of protocol
buffer field types which need extra maintenance. Currently these categories
are:
- Repeated scalar fields - These are all repeated fields which aren't
composite (e.g. they are of simple types like int32, string, etc).
- Repeated composite fields - Repeated fields which are composite. This
includes groups and nested messages.
"""
import collections.abc
import copy
import pickle
from typing import (
Any,
Iterable,
Iterator,
List,
MutableMapping,
MutableSequence,
NoReturn,
Optional,
Sequence,
TypeVar,
Union,
overload,
)
_T = TypeVar('_T')
_K = TypeVar('_K')
_V = TypeVar('_V')
class BaseContainer(Sequence[_T]):
"""Base container class."""
# Minimizes memory usage and disallows assignment to other attributes.
__slots__ = ['_message_listener', '_values']
def __init__(self, message_listener: Any) -> None:
"""
Args:
message_listener: A MessageListener implementation.
The RepeatedScalarFieldContainer will call this object's
Modified() method when it is modified.
"""
self._message_listener = message_listener
self._values = []
@overload
def __getitem__(self, key: int) -> _T:
...
@overload
def __getitem__(self, key: slice) -> List[_T]:
...
def __getitem__(self, key):
"""Retrieves item by the specified key."""
return self._values[key]
def __len__(self) -> int:
"""Returns the number of elements in the container."""
return len(self._values)
def __ne__(self, other: Any) -> bool:
"""Checks if another instance isn't equal to this one."""
# The concrete classes should define __eq__.
return not self == other
__hash__ = None
def __repr__(self) -> str:
return repr(self._values)
def sort(self, *args, **kwargs) -> None:
# Continue to support the old sort_function keyword argument.
# This is expected to be a rare occurrence, so use LBYL to avoid
# the overhead of actually catching KeyError.
if 'sort_function' in kwargs:
kwargs['cmp'] = kwargs.pop('sort_function')
self._values.sort(*args, **kwargs)
def reverse(self) -> None:
self._values.reverse()
# TODO: Remove this. BaseContainer does *not* conform to
# MutableSequence, only its subclasses do.
collections.abc.MutableSequence.register(BaseContainer)
class RepeatedScalarFieldContainer(BaseContainer[_T], MutableSequence[_T]):
"""Simple, type-checked, list-like container for holding repeated scalars."""
# Disallows assignment to other attributes.
__slots__ = ['_type_checker']
def __init__(
self,
message_listener: Any,
type_checker: Any,
) -> None:
"""Args:
message_listener: A MessageListener implementation. The
RepeatedScalarFieldContainer will call this object's Modified() method
when it is modified.
type_checker: A type_checkers.ValueChecker instance to run on elements
inserted into this container.
"""
super().__init__(message_listener)
self._type_checker = type_checker
def append(self, value: _T) -> None:
"""Appends an item to the list. Similar to list.append()."""
self._values.append(self._type_checker.CheckValue(value))
if not self._message_listener.dirty:
self._message_listener.Modified()
def insert(self, key: int, value: _T) -> None:
"""Inserts the item at the specified position. Similar to list.insert()."""
self._values.insert(key, self._type_checker.CheckValue(value))
if not self._message_listener.dirty:
self._message_listener.Modified()
def extend(self, elem_seq: Iterable[_T]) -> None:
"""Extends by appending the given iterable. Similar to list.extend()."""
elem_seq_iter = iter(elem_seq)
new_values = [self._type_checker.CheckValue(elem) for elem in elem_seq_iter]
if new_values:
self._values.extend(new_values)
self._message_listener.Modified()
def MergeFrom(
self,
other: Union['RepeatedScalarFieldContainer[_T]', Iterable[_T]],
) -> None:
"""Appends the contents of another repeated field of the same type to this
one. We do not check the types of the individual fields.
"""
self._values.extend(other)
self._message_listener.Modified()
def remove(self, elem: _T):
"""Removes an item from the list. Similar to list.remove()."""
self._values.remove(elem)
self._message_listener.Modified()
def pop(self, key: Optional[int] = -1) -> _T:
"""Removes and returns an item at a given index. Similar to list.pop()."""
value = self._values[key]
self.__delitem__(key)
return value
@overload
def __setitem__(self, key: int, value: _T) -> None:
...
@overload
def __setitem__(self, key: slice, value: Iterable[_T]) -> None:
...
def __setitem__(self, key, value) -> None:
"""Sets the item on the specified position."""
if isinstance(key, slice):
if key.step is not None:
raise ValueError('Extended slices not supported')
self._values[key] = map(self._type_checker.CheckValue, value)
self._message_listener.Modified()
else:
self._values[key] = self._type_checker.CheckValue(value)
self._message_listener.Modified()
def __delitem__(self, key: Union[int, slice]) -> None:
"""Deletes the item at the specified position."""
del self._values[key]
self._message_listener.Modified()
def __eq__(self, other: Any) -> bool:
"""Compares the current instance with another one."""
if self is other:
return True
# Special case for the same type which should be common and fast.
if isinstance(other, self.__class__):
return other._values == self._values
# We are presumably comparing against some other sequence type.
return other == self._values
def __deepcopy__(
self,
unused_memo: Any = None,
) -> 'RepeatedScalarFieldContainer[_T]':
clone = RepeatedScalarFieldContainer(
copy.deepcopy(self._message_listener), self._type_checker)
clone.MergeFrom(self)
return clone
def __reduce__(self, **kwargs) -> NoReturn:
raise pickle.PickleError(
"Can't pickle repeated scalar fields, convert to list first")
# TODO: Constrain T to be a subtype of Message.
class RepeatedCompositeFieldContainer(BaseContainer[_T], MutableSequence[_T]):
"""Simple, list-like container for holding repeated composite fields."""
# Disallows assignment to other attributes.
__slots__ = ['_message_descriptor']
def __init__(self, message_listener: Any, message_descriptor: Any) -> None:
"""
Note that we pass in a descriptor instead of the generated directly,
since at the time we construct a _RepeatedCompositeFieldContainer we
haven't yet necessarily initialized the type that will be contained in the
container.
Args:
message_listener: A MessageListener implementation.
The RepeatedCompositeFieldContainer will call this object's
Modified() method when it is modified.
message_descriptor: A Descriptor instance describing the protocol type
that should be present in this container. We'll use the
_concrete_class field of this descriptor when the client calls add().
"""
super().__init__(message_listener)
self._message_descriptor = message_descriptor
def add(self, **kwargs: Any) -> _T:
"""Adds a new element at the end of the list and returns it. Keyword
arguments may be used to initialize the element.
"""
new_element = self._message_descriptor._concrete_class(**kwargs)
new_element._SetListener(self._message_listener)
self._values.append(new_element)
if not self._message_listener.dirty:
self._message_listener.Modified()
return new_element
def append(self, value: _T) -> None:
"""Appends one element by copying the message."""
new_element = self._message_descriptor._concrete_class()
new_element._SetListener(self._message_listener)
new_element.CopyFrom(value)
self._values.append(new_element)
if not self._message_listener.dirty:
self._message_listener.Modified()
def insert(self, key: int, value: _T) -> None:
"""Inserts the item at the specified position by copying."""
new_element = self._message_descriptor._concrete_class()
new_element._SetListener(self._message_listener)
new_element.CopyFrom(value)
self._values.insert(key, new_element)
if not self._message_listener.dirty:
self._message_listener.Modified()
def extend(self, elem_seq: Iterable[_T]) -> None:
"""Extends by appending the given sequence of elements of the same type
as this one, copying each individual message.
"""
message_class = self._message_descriptor._concrete_class
listener = self._message_listener
values = self._values
for message in elem_seq:
new_element = message_class()
new_element._SetListener(listener)
new_element.MergeFrom(message)
values.append(new_element)
listener.Modified()
def MergeFrom(
self,
other: Union['RepeatedCompositeFieldContainer[_T]', Iterable[_T]],
) -> None:
"""Appends the contents of another repeated field of the same type to this
one, copying each individual message.
"""
self.extend(other)
def remove(self, elem: _T) -> None:
"""Removes an item from the list. Similar to list.remove()."""
self._values.remove(elem)
self._message_listener.Modified()
def pop(self, key: Optional[int] = -1) -> _T:
"""Removes and returns an item at a given index. Similar to list.pop()."""
value = self._values[key]
self.__delitem__(key)
return value
@overload
def __setitem__(self, key: int, value: _T) -> None:
...
@overload
def __setitem__(self, key: slice, value: Iterable[_T]) -> None:
...
def __setitem__(self, key, value):
# This method is implemented to make RepeatedCompositeFieldContainer
# structurally compatible with typing.MutableSequence. It is
# otherwise unsupported and will always raise an error.
raise TypeError(
f'{self.__class__.__name__} object does not support item assignment')
def __delitem__(self, key: Union[int, slice]) -> None:
"""Deletes the item at the specified position."""
del self._values[key]
self._message_listener.Modified()
def __eq__(self, other: Any) -> bool:
"""Compares the current instance with another one."""
if self is other:
return True
if not isinstance(other, self.__class__):
raise TypeError('Can only compare repeated composite fields against '
'other repeated composite fields.')
return self._values == other._values
class ScalarMap(MutableMapping[_K, _V]):
"""Simple, type-checked, dict-like container for holding repeated scalars."""
# Disallows assignment to other attributes.
__slots__ = ['_key_checker', '_value_checker', '_values', '_message_listener',
'_entry_descriptor']
def __init__(
self,
message_listener: Any,
key_checker: Any,
value_checker: Any,
entry_descriptor: Any,
) -> None:
"""
Args:
message_listener: A MessageListener implementation.
The ScalarMap will call this object's Modified() method when it
is modified.
key_checker: A type_checkers.ValueChecker instance to run on keys
inserted into this container.
value_checker: A type_checkers.ValueChecker instance to run on values
inserted into this container.
entry_descriptor: The MessageDescriptor of a map entry: key and value.
"""
self._message_listener = message_listener
self._key_checker = key_checker
self._value_checker = value_checker
self._entry_descriptor = entry_descriptor
self._values = {}
def __getitem__(self, key: _K) -> _V:
try:
return self._values[key]
except KeyError:
key = self._key_checker.CheckValue(key)
val = self._value_checker.DefaultValue()
self._values[key] = val
return val
def __contains__(self, item: _K) -> bool:
# We check the key's type to match the strong-typing flavor of the API.
# Also this makes it easier to match the behavior of the C++ implementation.
self._key_checker.CheckValue(item)
return item in self._values
@overload
def get(self, key: _K) -> Optional[_V]:
...
@overload
def get(self, key: _K, default: _T) -> Union[_V, _T]:
...
# We need to override this explicitly, because our defaultdict-like behavior
# will make the default implementation (from our base class) always insert
# the key.
def get(self, key, default=None):
if key in self:
return self[key]
else:
return default
def __setitem__(self, key: _K, value: _V) -> _T:
checked_key = self._key_checker.CheckValue(key)
checked_value = self._value_checker.CheckValue(value)
self._values[checked_key] = checked_value
self._message_listener.Modified()
def __delitem__(self, key: _K) -> None:
del self._values[key]
self._message_listener.Modified()
def __len__(self) -> int:
return len(self._values)
def __iter__(self) -> Iterator[_K]:
return iter(self._values)
def __repr__(self) -> str:
return repr(self._values)
def setdefault(self, key: _K, value: Optional[_V] = None) -> _V:
if value == None:
raise ValueError('The value for scalar map setdefault must be set.')
if key not in self._values:
self.__setitem__(key, value)
return self[key]
def MergeFrom(self, other: 'ScalarMap[_K, _V]') -> None:
self._values.update(other._values)
self._message_listener.Modified()
def InvalidateIterators(self) -> None:
# It appears that the only way to reliably invalidate iterators to
# self._values is to ensure that its size changes.
original = self._values
self._values = original.copy()
original[None] = None
# This is defined in the abstract base, but we can do it much more cheaply.
def clear(self) -> None:
self._values.clear()
self._message_listener.Modified()
def GetEntryClass(self) -> Any:
return self._entry_descriptor._concrete_class
class MessageMap(MutableMapping[_K, _V]):
"""Simple, type-checked, dict-like container for with submessage values."""
# Disallows assignment to other attributes.
__slots__ = ['_key_checker', '_values', '_message_listener',
'_message_descriptor', '_entry_descriptor']
def __init__(
self,
message_listener: Any,
message_descriptor: Any,
key_checker: Any,
entry_descriptor: Any,
) -> None:
"""
Args:
message_listener: A MessageListener implementation.
The ScalarMap will call this object's Modified() method when it
is modified.
key_checker: A type_checkers.ValueChecker instance to run on keys
inserted into this container.
value_checker: A type_checkers.ValueChecker instance to run on values
inserted into this container.
entry_descriptor: The MessageDescriptor of a map entry: key and value.
"""
self._message_listener = message_listener
self._message_descriptor = message_descriptor
self._key_checker = key_checker
self._entry_descriptor = entry_descriptor
self._values = {}
def __getitem__(self, key: _K) -> _V:
key = self._key_checker.CheckValue(key)
try:
return self._values[key]
except KeyError:
new_element = self._message_descriptor._concrete_class()
new_element._SetListener(self._message_listener)
self._values[key] = new_element
self._message_listener.Modified()
return new_element
def get_or_create(self, key: _K) -> _V:
"""get_or_create() is an alias for getitem (ie. map[key]).
Args:
key: The key to get or create in the map.
This is useful in cases where you want to be explicit that the call is
mutating the map. This can avoid lint errors for statements like this
that otherwise would appear to be pointless statements:
msg.my_map[key]
"""
return self[key]
@overload
def get(self, key: _K) -> Optional[_V]:
...
@overload
def get(self, key: _K, default: _T) -> Union[_V, _T]:
...
# We need to override this explicitly, because our defaultdict-like behavior
# will make the default implementation (from our base class) always insert
# the key.
def get(self, key, default=None):
if key in self:
return self[key]
else:
return default
def __contains__(self, item: _K) -> bool:
item = self._key_checker.CheckValue(item)
return item in self._values
def __setitem__(self, key: _K, value: _V) -> NoReturn:
raise ValueError('May not set values directly, call my_map[key].foo = 5')
def __delitem__(self, key: _K) -> None:
key = self._key_checker.CheckValue(key)
del self._values[key]
self._message_listener.Modified()
def __len__(self) -> int:
return len(self._values)
def __iter__(self) -> Iterator[_K]:
return iter(self._values)
def __repr__(self) -> str:
return repr(self._values)
def setdefault(self, key: _K, value: Optional[_V] = None) -> _V:
raise NotImplementedError(
'Set message map value directly is not supported, call'
' my_map[key].foo = 5'
)
def MergeFrom(self, other: 'MessageMap[_K, _V]') -> None:
# pylint: disable=protected-access
for key in other._values:
# According to documentation: "When parsing from the wire or when merging,
# if there are duplicate map keys the last key seen is used".
if key in self:
del self[key]
self[key].CopyFrom(other[key])
# self._message_listener.Modified() not required here, because
# mutations to submessages already propagate.
def InvalidateIterators(self) -> None:
# It appears that the only way to reliably invalidate iterators to
# self._values is to ensure that its size changes.
original = self._values
self._values = original.copy()
original[None] = None
# This is defined in the abstract base, but we can do it much more cheaply.
def clear(self) -> None:
self._values.clear()
self._message_listener.Modified()
def GetEntryClass(self) -> Any:
return self._entry_descriptor._concrete_class
class _UnknownField:
"""A parsed unknown field."""
# Disallows assignment to other attributes.
__slots__ = ['_field_number', '_wire_type', '_data']
def __init__(self, field_number, wire_type, data):
self._field_number = field_number
self._wire_type = wire_type
self._data = data
return
def __lt__(self, other):
# pylint: disable=protected-access
return self._field_number < other._field_number
def __eq__(self, other):
if self is other:
return True
# pylint: disable=protected-access
return (self._field_number == other._field_number and
self._wire_type == other._wire_type and
self._data == other._data)
class UnknownFieldRef: # pylint: disable=missing-class-docstring
def __init__(self, parent, index):
self._parent = parent
self._index = index
def _check_valid(self):
if not self._parent:
raise ValueError('UnknownField does not exist. '
'The parent message might be cleared.')
if self._index >= len(self._parent):
raise ValueError('UnknownField does not exist. '
'The parent message might be cleared.')
@property
def field_number(self):
self._check_valid()
# pylint: disable=protected-access
return self._parent._internal_get(self._index)._field_number
@property
def wire_type(self):
self._check_valid()
# pylint: disable=protected-access
return self._parent._internal_get(self._index)._wire_type
@property
def data(self):
self._check_valid()
# pylint: disable=protected-access
return self._parent._internal_get(self._index)._data
class UnknownFieldSet:
"""UnknownField container"""
# Disallows assignment to other attributes.
__slots__ = ['_values']
def __init__(self):
self._values = []
def __getitem__(self, index):
if self._values is None:
raise ValueError('UnknownFields does not exist. '
'The parent message might be cleared.')
size = len(self._values)
if index < 0:
index += size
if index < 0 or index >= size:
raise IndexError('index %d out of range'.index)
return UnknownFieldRef(self, index)
def _internal_get(self, index):
return self._values[index]
def __len__(self):
if self._values is None:
raise ValueError('UnknownFields does not exist. '
'The parent message might be cleared.')
return len(self._values)
def _add(self, field_number, wire_type, data):
unknown_field = _UnknownField(field_number, wire_type, data)
self._values.append(unknown_field)
return unknown_field
def __iter__(self):
for i in range(len(self)):
yield UnknownFieldRef(self, i)
def _extend(self, other):
if other is None:
return
# pylint: disable=protected-access
self._values.extend(other._values)
def __eq__(self, other):
if self is other:
return True
# Sort unknown fields because their order shouldn't
# affect equality test.
values = list(self._values)
if other is None:
return not values
values.sort()
# pylint: disable=protected-access
other_values = sorted(other._values)
return values == other_values
def _clear(self):
for value in self._values:
# pylint: disable=protected-access
if isinstance(value._data, UnknownFieldSet):
value._data._clear() # pylint: disable=protected-access
self._values = None
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,806 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Code for encoding protocol message primitives.
Contains the logic for encoding every logical protocol field type
into one of the 5 physical wire types.
This code is designed to push the Python interpreter's performance to the
limits.
The basic idea is that at startup time, for every field (i.e. every
FieldDescriptor) we construct two functions: a "sizer" and an "encoder". The
sizer takes a value of this field's type and computes its byte size. The
encoder takes a writer function and a value. It encodes the value into byte
strings and invokes the writer function to write those strings. Typically the
writer function is the write() method of a BytesIO.
We try to do as much work as possible when constructing the writer and the
sizer rather than when calling them. In particular:
* We copy any needed global functions to local variables, so that we do not need
to do costly global table lookups at runtime.
* Similarly, we try to do any attribute lookups at startup time if possible.
* Every field's tag is encoded to bytes at startup, since it can't change at
runtime.
* Whatever component of the field size we can compute at startup, we do.
* We *avoid* sharing code if doing so would make the code slower and not sharing
does not burden us too much. For example, encoders for repeated fields do
not just call the encoders for singular fields in a loop because this would
add an extra function call overhead for every loop iteration; instead, we
manually inline the single-value encoder into the loop.
* If a Python function lacks a return statement, Python actually generates
instructions to pop the result of the last statement off the stack, push
None onto the stack, and then return that. If we really don't care what
value is returned, then we can save two instructions by returning the
result of the last statement. It looks funny but it helps.
* We assume that type and bounds checking has happened at a higher level.
"""
__author__ = 'kenton@google.com (Kenton Varda)'
import struct
from google.protobuf.internal import wire_format
# This will overflow and thus become IEEE-754 "infinity". We would use
# "float('inf')" but it doesn't work on Windows pre-Python-2.6.
_POS_INF = 1e10000
_NEG_INF = -_POS_INF
def _VarintSize(value):
"""Compute the size of a varint value."""
if value <= 0x7f: return 1
if value <= 0x3fff: return 2
if value <= 0x1fffff: return 3
if value <= 0xfffffff: return 4
if value <= 0x7ffffffff: return 5
if value <= 0x3ffffffffff: return 6
if value <= 0x1ffffffffffff: return 7
if value <= 0xffffffffffffff: return 8
if value <= 0x7fffffffffffffff: return 9
return 10
def _SignedVarintSize(value):
"""Compute the size of a signed varint value."""
if value < 0: return 10
if value <= 0x7f: return 1
if value <= 0x3fff: return 2
if value <= 0x1fffff: return 3
if value <= 0xfffffff: return 4
if value <= 0x7ffffffff: return 5
if value <= 0x3ffffffffff: return 6
if value <= 0x1ffffffffffff: return 7
if value <= 0xffffffffffffff: return 8
if value <= 0x7fffffffffffffff: return 9
return 10
def _TagSize(field_number):
"""Returns the number of bytes required to serialize a tag with this field
number."""
# Just pass in type 0, since the type won't affect the tag+type size.
return _VarintSize(wire_format.PackTag(field_number, 0))
# --------------------------------------------------------------------
# In this section we define some generic sizers. Each of these functions
# takes parameters specific to a particular field type, e.g. int32 or fixed64.
# It returns another function which in turn takes parameters specific to a
# particular field, e.g. the field number and whether it is repeated or packed.
# Look at the next section to see how these are used.
def _SimpleSizer(compute_value_size):
"""A sizer which uses the function compute_value_size to compute the size of
each value. Typically compute_value_size is _VarintSize."""
def SpecificSizer(field_number, is_repeated, is_packed):
tag_size = _TagSize(field_number)
if is_packed:
local_VarintSize = _VarintSize
def PackedFieldSize(value):
result = 0
for element in value:
result += compute_value_size(element)
return result + local_VarintSize(result) + tag_size
return PackedFieldSize
elif is_repeated:
def RepeatedFieldSize(value):
result = tag_size * len(value)
for element in value:
result += compute_value_size(element)
return result
return RepeatedFieldSize
else:
def FieldSize(value):
return tag_size + compute_value_size(value)
return FieldSize
return SpecificSizer
def _ModifiedSizer(compute_value_size, modify_value):
"""Like SimpleSizer, but modify_value is invoked on each value before it is
passed to compute_value_size. modify_value is typically ZigZagEncode."""
def SpecificSizer(field_number, is_repeated, is_packed):
tag_size = _TagSize(field_number)
if is_packed:
local_VarintSize = _VarintSize
def PackedFieldSize(value):
result = 0
for element in value:
result += compute_value_size(modify_value(element))
return result + local_VarintSize(result) + tag_size
return PackedFieldSize
elif is_repeated:
def RepeatedFieldSize(value):
result = tag_size * len(value)
for element in value:
result += compute_value_size(modify_value(element))
return result
return RepeatedFieldSize
else:
def FieldSize(value):
return tag_size + compute_value_size(modify_value(value))
return FieldSize
return SpecificSizer
def _FixedSizer(value_size):
"""Like _SimpleSizer except for a fixed-size field. The input is the size
of one value."""
def SpecificSizer(field_number, is_repeated, is_packed):
tag_size = _TagSize(field_number)
if is_packed:
local_VarintSize = _VarintSize
def PackedFieldSize(value):
result = len(value) * value_size
return result + local_VarintSize(result) + tag_size
return PackedFieldSize
elif is_repeated:
element_size = value_size + tag_size
def RepeatedFieldSize(value):
return len(value) * element_size
return RepeatedFieldSize
else:
field_size = value_size + tag_size
def FieldSize(value):
return field_size
return FieldSize
return SpecificSizer
# ====================================================================
# Here we declare a sizer constructor for each field type. Each "sizer
# constructor" is a function that takes (field_number, is_repeated, is_packed)
# as parameters and returns a sizer, which in turn takes a field value as
# a parameter and returns its encoded size.
Int32Sizer = Int64Sizer = EnumSizer = _SimpleSizer(_SignedVarintSize)
UInt32Sizer = UInt64Sizer = _SimpleSizer(_VarintSize)
SInt32Sizer = SInt64Sizer = _ModifiedSizer(
_SignedVarintSize, wire_format.ZigZagEncode)
Fixed32Sizer = SFixed32Sizer = FloatSizer = _FixedSizer(4)
Fixed64Sizer = SFixed64Sizer = DoubleSizer = _FixedSizer(8)
BoolSizer = _FixedSizer(1)
def StringSizer(field_number, is_repeated, is_packed):
"""Returns a sizer for a string field."""
tag_size = _TagSize(field_number)
local_VarintSize = _VarintSize
local_len = len
assert not is_packed
if is_repeated:
def RepeatedFieldSize(value):
result = tag_size * len(value)
for element in value:
l = local_len(element.encode('utf-8'))
result += local_VarintSize(l) + l
return result
return RepeatedFieldSize
else:
def FieldSize(value):
l = local_len(value.encode('utf-8'))
return tag_size + local_VarintSize(l) + l
return FieldSize
def BytesSizer(field_number, is_repeated, is_packed):
"""Returns a sizer for a bytes field."""
tag_size = _TagSize(field_number)
local_VarintSize = _VarintSize
local_len = len
assert not is_packed
if is_repeated:
def RepeatedFieldSize(value):
result = tag_size * len(value)
for element in value:
l = local_len(element)
result += local_VarintSize(l) + l
return result
return RepeatedFieldSize
else:
def FieldSize(value):
l = local_len(value)
return tag_size + local_VarintSize(l) + l
return FieldSize
def GroupSizer(field_number, is_repeated, is_packed):
"""Returns a sizer for a group field."""
tag_size = _TagSize(field_number) * 2
assert not is_packed
if is_repeated:
def RepeatedFieldSize(value):
result = tag_size * len(value)
for element in value:
result += element.ByteSize()
return result
return RepeatedFieldSize
else:
def FieldSize(value):
return tag_size + value.ByteSize()
return FieldSize
def MessageSizer(field_number, is_repeated, is_packed):
"""Returns a sizer for a message field."""
tag_size = _TagSize(field_number)
local_VarintSize = _VarintSize
assert not is_packed
if is_repeated:
def RepeatedFieldSize(value):
result = tag_size * len(value)
for element in value:
l = element.ByteSize()
result += local_VarintSize(l) + l
return result
return RepeatedFieldSize
else:
def FieldSize(value):
l = value.ByteSize()
return tag_size + local_VarintSize(l) + l
return FieldSize
# --------------------------------------------------------------------
# MessageSet is special: it needs custom logic to compute its size properly.
def MessageSetItemSizer(field_number):
"""Returns a sizer for extensions of MessageSet.
The message set message looks like this:
message MessageSet {
repeated group Item = 1 {
required int32 type_id = 2;
required string message = 3;
}
}
"""
static_size = (_TagSize(1) * 2 + _TagSize(2) + _VarintSize(field_number) +
_TagSize(3))
local_VarintSize = _VarintSize
def FieldSize(value):
l = value.ByteSize()
return static_size + local_VarintSize(l) + l
return FieldSize
# --------------------------------------------------------------------
# Map is special: it needs custom logic to compute its size properly.
def MapSizer(field_descriptor, is_message_map):
"""Returns a sizer for a map field."""
# Can't look at field_descriptor.message_type._concrete_class because it may
# not have been initialized yet.
message_type = field_descriptor.message_type
message_sizer = MessageSizer(field_descriptor.number, False, False)
def FieldSize(map_value):
total = 0
for key in map_value:
value = map_value[key]
# It's wasteful to create the messages and throw them away one second
# later since we'll do the same for the actual encode. But there's not an
# obvious way to avoid this within the current design without tons of code
# duplication. For message map, value.ByteSize() should be called to
# update the status.
entry_msg = message_type._concrete_class(key=key, value=value)
total += message_sizer(entry_msg)
if is_message_map:
value.ByteSize()
return total
return FieldSize
# ====================================================================
# Encoders!
def _VarintEncoder():
"""Return an encoder for a basic varint value (does not include tag)."""
local_int2byte = struct.Struct('>B').pack
def EncodeVarint(write, value, unused_deterministic=None):
bits = value & 0x7f
value >>= 7
while value:
write(local_int2byte(0x80|bits))
bits = value & 0x7f
value >>= 7
return write(local_int2byte(bits))
return EncodeVarint
def _SignedVarintEncoder():
"""Return an encoder for a basic signed varint value (does not include
tag)."""
local_int2byte = struct.Struct('>B').pack
def EncodeSignedVarint(write, value, unused_deterministic=None):
if value < 0:
value += (1 << 64)
bits = value & 0x7f
value >>= 7
while value:
write(local_int2byte(0x80|bits))
bits = value & 0x7f
value >>= 7
return write(local_int2byte(bits))
return EncodeSignedVarint
_EncodeVarint = _VarintEncoder()
_EncodeSignedVarint = _SignedVarintEncoder()
def _VarintBytes(value):
"""Encode the given integer as a varint and return the bytes. This is only
called at startup time so it doesn't need to be fast."""
pieces = []
_EncodeVarint(pieces.append, value, True)
return b"".join(pieces)
def TagBytes(field_number, wire_type):
"""Encode the given tag and return the bytes. Only called at startup."""
return bytes(_VarintBytes(wire_format.PackTag(field_number, wire_type)))
# --------------------------------------------------------------------
# As with sizers (see above), we have a number of common encoder
# implementations.
def _SimpleEncoder(wire_type, encode_value, compute_value_size):
"""Return a constructor for an encoder for fields of a particular type.
Args:
wire_type: The field's wire type, for encoding tags.
encode_value: A function which encodes an individual value, e.g.
_EncodeVarint().
compute_value_size: A function which computes the size of an individual
value, e.g. _VarintSize().
"""
def SpecificEncoder(field_number, is_repeated, is_packed):
if is_packed:
tag_bytes = TagBytes(field_number, wire_format.WIRETYPE_LENGTH_DELIMITED)
local_EncodeVarint = _EncodeVarint
def EncodePackedField(write, value, deterministic):
write(tag_bytes)
size = 0
for element in value:
size += compute_value_size(element)
local_EncodeVarint(write, size, deterministic)
for element in value:
encode_value(write, element, deterministic)
return EncodePackedField
elif is_repeated:
tag_bytes = TagBytes(field_number, wire_type)
def EncodeRepeatedField(write, value, deterministic):
for element in value:
write(tag_bytes)
encode_value(write, element, deterministic)
return EncodeRepeatedField
else:
tag_bytes = TagBytes(field_number, wire_type)
def EncodeField(write, value, deterministic):
write(tag_bytes)
return encode_value(write, value, deterministic)
return EncodeField
return SpecificEncoder
def _ModifiedEncoder(wire_type, encode_value, compute_value_size, modify_value):
"""Like SimpleEncoder but additionally invokes modify_value on every value
before passing it to encode_value. Usually modify_value is ZigZagEncode."""
def SpecificEncoder(field_number, is_repeated, is_packed):
if is_packed:
tag_bytes = TagBytes(field_number, wire_format.WIRETYPE_LENGTH_DELIMITED)
local_EncodeVarint = _EncodeVarint
def EncodePackedField(write, value, deterministic):
write(tag_bytes)
size = 0
for element in value:
size += compute_value_size(modify_value(element))
local_EncodeVarint(write, size, deterministic)
for element in value:
encode_value(write, modify_value(element), deterministic)
return EncodePackedField
elif is_repeated:
tag_bytes = TagBytes(field_number, wire_type)
def EncodeRepeatedField(write, value, deterministic):
for element in value:
write(tag_bytes)
encode_value(write, modify_value(element), deterministic)
return EncodeRepeatedField
else:
tag_bytes = TagBytes(field_number, wire_type)
def EncodeField(write, value, deterministic):
write(tag_bytes)
return encode_value(write, modify_value(value), deterministic)
return EncodeField
return SpecificEncoder
def _StructPackEncoder(wire_type, format):
"""Return a constructor for an encoder for a fixed-width field.
Args:
wire_type: The field's wire type, for encoding tags.
format: The format string to pass to struct.pack().
"""
value_size = struct.calcsize(format)
def SpecificEncoder(field_number, is_repeated, is_packed):
local_struct_pack = struct.pack
if is_packed:
tag_bytes = TagBytes(field_number, wire_format.WIRETYPE_LENGTH_DELIMITED)
local_EncodeVarint = _EncodeVarint
def EncodePackedField(write, value, deterministic):
write(tag_bytes)
local_EncodeVarint(write, len(value) * value_size, deterministic)
for element in value:
write(local_struct_pack(format, element))
return EncodePackedField
elif is_repeated:
tag_bytes = TagBytes(field_number, wire_type)
def EncodeRepeatedField(write, value, unused_deterministic=None):
for element in value:
write(tag_bytes)
write(local_struct_pack(format, element))
return EncodeRepeatedField
else:
tag_bytes = TagBytes(field_number, wire_type)
def EncodeField(write, value, unused_deterministic=None):
write(tag_bytes)
return write(local_struct_pack(format, value))
return EncodeField
return SpecificEncoder
def _FloatingPointEncoder(wire_type, format):
"""Return a constructor for an encoder for float fields.
This is like StructPackEncoder, but catches errors that may be due to
passing non-finite floating-point values to struct.pack, and makes a
second attempt to encode those values.
Args:
wire_type: The field's wire type, for encoding tags.
format: The format string to pass to struct.pack().
"""
value_size = struct.calcsize(format)
if value_size == 4:
def EncodeNonFiniteOrRaise(write, value):
# Remember that the serialized form uses little-endian byte order.
if value == _POS_INF:
write(b'\x00\x00\x80\x7F')
elif value == _NEG_INF:
write(b'\x00\x00\x80\xFF')
elif value != value: # NaN
write(b'\x00\x00\xC0\x7F')
else:
raise
elif value_size == 8:
def EncodeNonFiniteOrRaise(write, value):
if value == _POS_INF:
write(b'\x00\x00\x00\x00\x00\x00\xF0\x7F')
elif value == _NEG_INF:
write(b'\x00\x00\x00\x00\x00\x00\xF0\xFF')
elif value != value: # NaN
write(b'\x00\x00\x00\x00\x00\x00\xF8\x7F')
else:
raise
else:
raise ValueError('Can\'t encode floating-point values that are '
'%d bytes long (only 4 or 8)' % value_size)
def SpecificEncoder(field_number, is_repeated, is_packed):
local_struct_pack = struct.pack
if is_packed:
tag_bytes = TagBytes(field_number, wire_format.WIRETYPE_LENGTH_DELIMITED)
local_EncodeVarint = _EncodeVarint
def EncodePackedField(write, value, deterministic):
write(tag_bytes)
local_EncodeVarint(write, len(value) * value_size, deterministic)
for element in value:
# This try/except block is going to be faster than any code that
# we could write to check whether element is finite.
try:
write(local_struct_pack(format, element))
except SystemError:
EncodeNonFiniteOrRaise(write, element)
return EncodePackedField
elif is_repeated:
tag_bytes = TagBytes(field_number, wire_type)
def EncodeRepeatedField(write, value, unused_deterministic=None):
for element in value:
write(tag_bytes)
try:
write(local_struct_pack(format, element))
except SystemError:
EncodeNonFiniteOrRaise(write, element)
return EncodeRepeatedField
else:
tag_bytes = TagBytes(field_number, wire_type)
def EncodeField(write, value, unused_deterministic=None):
write(tag_bytes)
try:
write(local_struct_pack(format, value))
except SystemError:
EncodeNonFiniteOrRaise(write, value)
return EncodeField
return SpecificEncoder
# ====================================================================
# Here we declare an encoder constructor for each field type. These work
# very similarly to sizer constructors, described earlier.
Int32Encoder = Int64Encoder = EnumEncoder = _SimpleEncoder(
wire_format.WIRETYPE_VARINT, _EncodeSignedVarint, _SignedVarintSize)
UInt32Encoder = UInt64Encoder = _SimpleEncoder(
wire_format.WIRETYPE_VARINT, _EncodeVarint, _VarintSize)
SInt32Encoder = SInt64Encoder = _ModifiedEncoder(
wire_format.WIRETYPE_VARINT, _EncodeVarint, _VarintSize,
wire_format.ZigZagEncode)
# Note that Python conveniently guarantees that when using the '<' prefix on
# formats, they will also have the same size across all platforms (as opposed
# to without the prefix, where their sizes depend on the C compiler's basic
# type sizes).
Fixed32Encoder = _StructPackEncoder(wire_format.WIRETYPE_FIXED32, '<I')
Fixed64Encoder = _StructPackEncoder(wire_format.WIRETYPE_FIXED64, '<Q')
SFixed32Encoder = _StructPackEncoder(wire_format.WIRETYPE_FIXED32, '<i')
SFixed64Encoder = _StructPackEncoder(wire_format.WIRETYPE_FIXED64, '<q')
FloatEncoder = _FloatingPointEncoder(wire_format.WIRETYPE_FIXED32, '<f')
DoubleEncoder = _FloatingPointEncoder(wire_format.WIRETYPE_FIXED64, '<d')
def BoolEncoder(field_number, is_repeated, is_packed):
"""Returns an encoder for a boolean field."""
false_byte = b'\x00'
true_byte = b'\x01'
if is_packed:
tag_bytes = TagBytes(field_number, wire_format.WIRETYPE_LENGTH_DELIMITED)
local_EncodeVarint = _EncodeVarint
def EncodePackedField(write, value, deterministic):
write(tag_bytes)
local_EncodeVarint(write, len(value), deterministic)
for element in value:
if element:
write(true_byte)
else:
write(false_byte)
return EncodePackedField
elif is_repeated:
tag_bytes = TagBytes(field_number, wire_format.WIRETYPE_VARINT)
def EncodeRepeatedField(write, value, unused_deterministic=None):
for element in value:
write(tag_bytes)
if element:
write(true_byte)
else:
write(false_byte)
return EncodeRepeatedField
else:
tag_bytes = TagBytes(field_number, wire_format.WIRETYPE_VARINT)
def EncodeField(write, value, unused_deterministic=None):
write(tag_bytes)
if value:
return write(true_byte)
return write(false_byte)
return EncodeField
def StringEncoder(field_number, is_repeated, is_packed):
"""Returns an encoder for a string field."""
tag = TagBytes(field_number, wire_format.WIRETYPE_LENGTH_DELIMITED)
local_EncodeVarint = _EncodeVarint
local_len = len
assert not is_packed
if is_repeated:
def EncodeRepeatedField(write, value, deterministic):
for element in value:
encoded = element.encode('utf-8')
write(tag)
local_EncodeVarint(write, local_len(encoded), deterministic)
write(encoded)
return EncodeRepeatedField
else:
def EncodeField(write, value, deterministic):
encoded = value.encode('utf-8')
write(tag)
local_EncodeVarint(write, local_len(encoded), deterministic)
return write(encoded)
return EncodeField
def BytesEncoder(field_number, is_repeated, is_packed):
"""Returns an encoder for a bytes field."""
tag = TagBytes(field_number, wire_format.WIRETYPE_LENGTH_DELIMITED)
local_EncodeVarint = _EncodeVarint
local_len = len
assert not is_packed
if is_repeated:
def EncodeRepeatedField(write, value, deterministic):
for element in value:
write(tag)
local_EncodeVarint(write, local_len(element), deterministic)
write(element)
return EncodeRepeatedField
else:
def EncodeField(write, value, deterministic):
write(tag)
local_EncodeVarint(write, local_len(value), deterministic)
return write(value)
return EncodeField
def GroupEncoder(field_number, is_repeated, is_packed):
"""Returns an encoder for a group field."""
start_tag = TagBytes(field_number, wire_format.WIRETYPE_START_GROUP)
end_tag = TagBytes(field_number, wire_format.WIRETYPE_END_GROUP)
assert not is_packed
if is_repeated:
def EncodeRepeatedField(write, value, deterministic):
for element in value:
write(start_tag)
element._InternalSerialize(write, deterministic)
write(end_tag)
return EncodeRepeatedField
else:
def EncodeField(write, value, deterministic):
write(start_tag)
value._InternalSerialize(write, deterministic)
return write(end_tag)
return EncodeField
def MessageEncoder(field_number, is_repeated, is_packed):
"""Returns an encoder for a message field."""
tag = TagBytes(field_number, wire_format.WIRETYPE_LENGTH_DELIMITED)
local_EncodeVarint = _EncodeVarint
assert not is_packed
if is_repeated:
def EncodeRepeatedField(write, value, deterministic):
for element in value:
write(tag)
local_EncodeVarint(write, element.ByteSize(), deterministic)
element._InternalSerialize(write, deterministic)
return EncodeRepeatedField
else:
def EncodeField(write, value, deterministic):
write(tag)
local_EncodeVarint(write, value.ByteSize(), deterministic)
return value._InternalSerialize(write, deterministic)
return EncodeField
# --------------------------------------------------------------------
# As before, MessageSet is special.
def MessageSetItemEncoder(field_number):
"""Encoder for extensions of MessageSet.
The message set message looks like this:
message MessageSet {
repeated group Item = 1 {
required int32 type_id = 2;
required string message = 3;
}
}
"""
start_bytes = b"".join([
TagBytes(1, wire_format.WIRETYPE_START_GROUP),
TagBytes(2, wire_format.WIRETYPE_VARINT),
_VarintBytes(field_number),
TagBytes(3, wire_format.WIRETYPE_LENGTH_DELIMITED)])
end_bytes = TagBytes(1, wire_format.WIRETYPE_END_GROUP)
local_EncodeVarint = _EncodeVarint
def EncodeField(write, value, deterministic):
write(start_bytes)
local_EncodeVarint(write, value.ByteSize(), deterministic)
value._InternalSerialize(write, deterministic)
return write(end_bytes)
return EncodeField
# --------------------------------------------------------------------
# As before, Map is special.
def MapEncoder(field_descriptor):
"""Encoder for extensions of MessageSet.
Maps always have a wire format like this:
message MapEntry {
key_type key = 1;
value_type value = 2;
}
repeated MapEntry map = N;
"""
# Can't look at field_descriptor.message_type._concrete_class because it may
# not have been initialized yet.
message_type = field_descriptor.message_type
encode_message = MessageEncoder(field_descriptor.number, False, False)
def EncodeField(write, value, deterministic):
value_keys = sorted(value.keys()) if deterministic else value
for key in value_keys:
entry_msg = message_type._concrete_class(key=key, value=value[key])
encode_message(write, entry_msg, deterministic)
return EncodeField
@@ -0,0 +1,112 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""A simple wrapper around enum types to expose utility functions.
Instances are created as properties with the same name as the enum they wrap
on proto classes. For usage, see:
reflection_test.py
"""
import sys
__author__ = 'rabsatt@google.com (Kevin Rabsatt)'
class EnumTypeWrapper(object):
"""A utility for finding the names of enum values."""
DESCRIPTOR = None
# This is a type alias, which mypy typing stubs can type as
# a genericized parameter constrained to an int, allowing subclasses
# to be typed with more constraint in .pyi stubs
# Eg.
# def MyGeneratedEnum(Message):
# ValueType = NewType('ValueType', int)
# def Name(self, number: MyGeneratedEnum.ValueType) -> str
ValueType = int
def __init__(self, enum_type):
"""Inits EnumTypeWrapper with an EnumDescriptor."""
self._enum_type = enum_type
self.DESCRIPTOR = enum_type # pylint: disable=invalid-name
def Name(self, number): # pylint: disable=invalid-name
"""Returns a string containing the name of an enum value."""
try:
return self._enum_type.values_by_number[number].name
except KeyError:
pass # fall out to break exception chaining
if not isinstance(number, int):
raise TypeError(
'Enum value for {} must be an int, but got {} {!r}.'.format(
self._enum_type.name, type(number), number))
else:
# repr here to handle the odd case when you pass in a boolean.
raise ValueError('Enum {} has no name defined for value {!r}'.format(
self._enum_type.name, number))
def Value(self, name): # pylint: disable=invalid-name
"""Returns the value corresponding to the given enum name."""
try:
return self._enum_type.values_by_name[name].number
except KeyError:
pass # fall out to break exception chaining
raise ValueError('Enum {} has no value defined for name {!r}'.format(
self._enum_type.name, name))
def keys(self):
"""Return a list of the string names in the enum.
Returns:
A list of strs, in the order they were defined in the .proto file.
"""
return [value_descriptor.name
for value_descriptor in self._enum_type.values]
def values(self):
"""Return a list of the integer values in the enum.
Returns:
A list of ints, in the order they were defined in the .proto file.
"""
return [value_descriptor.number
for value_descriptor in self._enum_type.values]
def items(self):
"""Return a list of the (name, value) pairs of the enum.
Returns:
A list of (str, int) pairs, in the order they were defined
in the .proto file.
"""
return [(value_descriptor.name, value_descriptor.number)
for value_descriptor in self._enum_type.values]
def __getattr__(self, name):
"""Returns the value corresponding to the given enum name."""
try:
return super(
EnumTypeWrapper,
self).__getattribute__('_enum_type').values_by_name[name].number
except KeyError:
pass # fall out to break exception chaining
raise AttributeError('Enum {} has no value defined for name {!r}'.format(
self._enum_type.name, name))
def __or__(self, other):
"""Returns the union type of self and other."""
if sys.version_info >= (3, 10):
return type(self) | other
else:
raise NotImplementedError(
'You may not use | on EnumTypes (or classes) below python 3.10'
)
@@ -0,0 +1,194 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Contains _ExtensionDict class to represent extensions.
"""
from google.protobuf.internal import type_checkers
from google.protobuf.descriptor import FieldDescriptor
def _VerifyExtensionHandle(message, extension_handle):
"""Verify that the given extension handle is valid."""
if not isinstance(extension_handle, FieldDescriptor):
raise KeyError('HasExtension() expects an extension handle, got: %s' %
extension_handle)
if not extension_handle.is_extension:
raise KeyError('"%s" is not an extension.' % extension_handle.full_name)
if not extension_handle.containing_type:
raise KeyError('"%s" is missing a containing_type.'
% extension_handle.full_name)
if extension_handle.containing_type is not message.DESCRIPTOR:
raise KeyError('Extension "%s" extends message type "%s", but this '
'message is of type "%s".' %
(extension_handle.full_name,
extension_handle.containing_type.full_name,
message.DESCRIPTOR.full_name))
# TODO: Unify error handling of "unknown extension" crap.
# TODO: Support iteritems()-style iteration over all
# extensions with the "has" bits turned on?
class _ExtensionDict(object):
"""Dict-like container for Extension fields on proto instances.
Note that in all cases we expect extension handles to be
FieldDescriptors.
"""
def __init__(self, extended_message):
"""
Args:
extended_message: Message instance for which we are the Extensions dict.
"""
self._extended_message = extended_message
def __getitem__(self, extension_handle):
"""Returns the current value of the given extension handle."""
_VerifyExtensionHandle(self._extended_message, extension_handle)
result = self._extended_message._fields.get(extension_handle)
if result is not None:
return result
if extension_handle.is_repeated:
result = extension_handle._default_constructor(self._extended_message)
elif extension_handle.cpp_type == FieldDescriptor.CPPTYPE_MESSAGE:
message_type = extension_handle.message_type
if not hasattr(message_type, '_concrete_class'):
# pylint: disable=g-import-not-at-top
from google.protobuf import message_factory
message_factory.GetMessageClass(message_type)
if not hasattr(extension_handle.message_type, '_concrete_class'):
from google.protobuf import message_factory
message_factory.GetMessageClass(extension_handle.message_type)
result = extension_handle.message_type._concrete_class()
try:
result._SetListener(self._extended_message._listener_for_children)
except ReferenceError:
pass
else:
# Singular scalar -- just return the default without inserting into the
# dict.
return extension_handle.default_value
# Atomically check if another thread has preempted us and, if not, swap
# in the new object we just created. If someone has preempted us, we
# take that object and discard ours.
# WARNING: We are relying on setdefault() being atomic. This is true
# in CPython but we haven't investigated others. This warning appears
# in several other locations in this file.
result = self._extended_message._fields.setdefault(
extension_handle, result)
return result
def __eq__(self, other):
if not isinstance(other, self.__class__):
return False
my_fields = self._extended_message.ListFields()
other_fields = other._extended_message.ListFields()
# Get rid of non-extension fields.
my_fields = [field for field in my_fields if field.is_extension]
other_fields = [field for field in other_fields if field.is_extension]
return my_fields == other_fields
def __ne__(self, other):
return not self == other
def __len__(self):
fields = self._extended_message.ListFields()
# Get rid of non-extension fields.
extension_fields = [field for field in fields if field[0].is_extension]
return len(extension_fields)
def __hash__(self):
raise TypeError('unhashable object')
# Note that this is only meaningful for non-repeated, scalar extension
# fields. Note also that we may have to call _Modified() when we do
# successfully set a field this way, to set any necessary "has" bits in the
# ancestors of the extended message.
def __setitem__(self, extension_handle, value):
"""If extension_handle specifies a non-repeated, scalar extension
field, sets the value of that field.
"""
_VerifyExtensionHandle(self._extended_message, extension_handle)
if (extension_handle.is_repeated or
extension_handle.cpp_type == FieldDescriptor.CPPTYPE_MESSAGE):
raise TypeError(
'Cannot assign to extension "%s" because it is a repeated or '
'composite type.' % extension_handle.full_name)
# It's slightly wasteful to lookup the type checker each time,
# but we expect this to be a vanishingly uncommon case anyway.
type_checker = type_checkers.GetTypeChecker(extension_handle)
# pylint: disable=protected-access
self._extended_message._fields[extension_handle] = (
type_checker.CheckValue(value))
self._extended_message._Modified()
def __delitem__(self, extension_handle):
self._extended_message.ClearExtension(extension_handle)
def _FindExtensionByName(self, name):
"""Tries to find a known extension with the specified name.
Args:
name: Extension full name.
Returns:
Extension field descriptor.
"""
descriptor = self._extended_message.DESCRIPTOR
extensions = descriptor.file.pool._extensions_by_name[descriptor]
return extensions.get(name, None)
def _FindExtensionByNumber(self, number):
"""Tries to find a known extension with the field number.
Args:
number: Extension field number.
Returns:
Extension field descriptor.
"""
descriptor = self._extended_message.DESCRIPTOR
extensions = descriptor.file.pool._extensions_by_number[descriptor]
return extensions.get(number, None)
def __iter__(self):
# Return a generator over the populated extension fields
return (f[0] for f in self._extended_message.ListFields()
if f[0].is_extension)
def __contains__(self, extension_handle):
_VerifyExtensionHandle(self._extended_message, extension_handle)
if extension_handle not in self._extended_message._fields:
return False
if extension_handle.is_repeated:
return bool(self._extended_message._fields.get(extension_handle))
if extension_handle.cpp_type == FieldDescriptor.CPPTYPE_MESSAGE:
value = self._extended_message._fields.get(extension_handle)
# pylint: disable=protected-access
return value is not None and value._is_present_in_parent
return True
@@ -0,0 +1,312 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Contains FieldMask class."""
from google.protobuf.descriptor import FieldDescriptor
class FieldMask(object):
"""Class for FieldMask message type."""
__slots__ = ()
def ToJsonString(self):
"""Converts FieldMask to string according to proto3 JSON spec."""
camelcase_paths = []
for path in self.paths:
camelcase_paths.append(_SnakeCaseToCamelCase(path))
return ','.join(camelcase_paths)
def FromJsonString(self, value):
"""Converts string to FieldMask according to proto3 JSON spec."""
if not isinstance(value, str):
raise ValueError('FieldMask JSON value not a string: {!r}'.format(value))
self.Clear()
if value:
for path in value.split(','):
self.paths.append(_CamelCaseToSnakeCase(path))
def IsValidForDescriptor(self, message_descriptor):
"""Checks whether the FieldMask is valid for Message Descriptor."""
for path in self.paths:
if not _IsValidPath(message_descriptor, path):
return False
return True
def AllFieldsFromDescriptor(self, message_descriptor):
"""Gets all direct fields of Message Descriptor to FieldMask."""
self.Clear()
for field in message_descriptor.fields:
self.paths.append(field.name)
def CanonicalFormFromMask(self, mask):
"""Converts a FieldMask to the canonical form.
Removes paths that are covered by another path. For example,
"foo.bar" is covered by "foo" and will be removed if "foo"
is also in the FieldMask. Then sorts all paths in alphabetical order.
Args:
mask: The original FieldMask to be converted.
"""
tree = _FieldMaskTree(mask)
tree.ToFieldMask(self)
def Union(self, mask1, mask2):
"""Merges mask1 and mask2 into this FieldMask."""
_CheckFieldMaskMessage(mask1)
_CheckFieldMaskMessage(mask2)
tree = _FieldMaskTree(mask1)
tree.MergeFromFieldMask(mask2)
tree.ToFieldMask(self)
def Intersect(self, mask1, mask2):
"""Intersects mask1 and mask2 into this FieldMask."""
_CheckFieldMaskMessage(mask1)
_CheckFieldMaskMessage(mask2)
tree = _FieldMaskTree(mask1)
intersection = _FieldMaskTree()
for path in mask2.paths:
tree.IntersectPath(path, intersection)
intersection.ToFieldMask(self)
def MergeMessage(
self, source, destination,
replace_message_field=False, replace_repeated_field=False):
"""Merges fields specified in FieldMask from source to destination.
Args:
source: Source message.
destination: The destination message to be merged into.
replace_message_field: Replace message field if True. Merge message
field if False.
replace_repeated_field: Replace repeated field if True. Append
elements of repeated field if False.
"""
tree = _FieldMaskTree(self)
tree.MergeMessage(
source, destination, replace_message_field, replace_repeated_field)
def _IsValidPath(message_descriptor, path):
"""Checks whether the path is valid for Message Descriptor."""
parts = path.split('.')
last = parts.pop()
for name in parts:
field = message_descriptor.fields_by_name.get(name)
if (field is None or
field.is_repeated or
field.type != FieldDescriptor.TYPE_MESSAGE):
return False
message_descriptor = field.message_type
return last in message_descriptor.fields_by_name
def _CheckFieldMaskMessage(message):
"""Raises ValueError if message is not a FieldMask."""
message_descriptor = message.DESCRIPTOR
if (message_descriptor.name != 'FieldMask' or
message_descriptor.file.name != 'google/protobuf/field_mask.proto'):
raise ValueError('Message {0} is not a FieldMask.'.format(
message_descriptor.full_name))
def _SnakeCaseToCamelCase(path_name):
"""Converts a path name from snake_case to camelCase."""
result = []
after_underscore = False
for c in path_name:
if c.isupper():
raise ValueError(
'Fail to print FieldMask to Json string: Path name '
'{0} must not contain uppercase letters.'.format(path_name))
if after_underscore:
if c.islower():
result.append(c.upper())
after_underscore = False
else:
raise ValueError(
'Fail to print FieldMask to Json string: The '
'character after a "_" must be a lowercase letter '
'in path name {0}.'.format(path_name))
elif c == '_':
after_underscore = True
else:
result += c
if after_underscore:
raise ValueError('Fail to print FieldMask to Json string: Trailing "_" '
'in path name {0}.'.format(path_name))
return ''.join(result)
def _CamelCaseToSnakeCase(path_name):
"""Converts a field name from camelCase to snake_case."""
result = []
for c in path_name:
if c == '_':
raise ValueError('Fail to parse FieldMask: Path name '
'{0} must not contain "_"s.'.format(path_name))
if c.isupper():
result += '_'
result += c.lower()
else:
result += c
return ''.join(result)
class _FieldMaskTree(object):
"""Represents a FieldMask in a tree structure.
For example, given a FieldMask "foo.bar,foo.baz,bar.baz",
the FieldMaskTree will be:
[_root] -+- foo -+- bar
| |
| +- baz
|
+- bar --- baz
In the tree, each leaf node represents a field path.
"""
__slots__ = ('_root',)
def __init__(self, field_mask=None):
"""Initializes the tree by FieldMask."""
self._root = {}
if field_mask:
self.MergeFromFieldMask(field_mask)
def MergeFromFieldMask(self, field_mask):
"""Merges a FieldMask to the tree."""
for path in field_mask.paths:
self.AddPath(path)
def AddPath(self, path):
"""Adds a field path into the tree.
If the field path to add is a sub-path of an existing field path
in the tree (i.e., a leaf node), it means the tree already matches
the given path so nothing will be added to the tree. If the path
matches an existing non-leaf node in the tree, that non-leaf node
will be turned into a leaf node with all its children removed because
the path matches all the node's children. Otherwise, a new path will
be added.
Args:
path: The field path to add.
"""
node = self._root
for name in path.split('.'):
if name not in node:
node[name] = {}
elif not node[name]:
# Pre-existing empty node implies we already have this entire tree.
return
node = node[name]
# Remove any sub-trees we might have had.
node.clear()
def ToFieldMask(self, field_mask):
"""Converts the tree to a FieldMask."""
field_mask.Clear()
_AddFieldPaths(self._root, '', field_mask)
def IntersectPath(self, path, intersection):
"""Calculates the intersection part of a field path with this tree.
Args:
path: The field path to calculates.
intersection: The out tree to record the intersection part.
"""
node = self._root
for name in path.split('.'):
if name not in node:
return
elif not node[name]:
intersection.AddPath(path)
return
node = node[name]
intersection.AddLeafNodes(path, node)
def AddLeafNodes(self, prefix, node):
"""Adds leaf nodes begin with prefix to this tree."""
if not node:
self.AddPath(prefix)
for name in node:
child_path = prefix + '.' + name
self.AddLeafNodes(child_path, node[name])
def MergeMessage(
self, source, destination,
replace_message, replace_repeated):
"""Merge all fields specified by this tree from source to destination."""
_MergeMessage(
self._root, source, destination, replace_message, replace_repeated)
def _StrConvert(value):
"""Converts value to str if it is not."""
# This file is imported by c extension and some methods like ClearField
# requires string for the field name. py2/py3 has different text
# type and may use unicode.
if not isinstance(value, str):
return value.encode('utf-8')
return value
def _MergeMessage(
node, source, destination, replace_message, replace_repeated):
"""Merge all fields specified by a sub-tree from source to destination."""
source_descriptor = source.DESCRIPTOR
for name in node:
child = node[name]
field = source_descriptor.fields_by_name[name]
if field is None:
raise ValueError('Error: Can\'t find field {0} in message {1}.'.format(
name, source_descriptor.full_name))
if child:
# Sub-paths are only allowed for singular message fields.
if (field.is_repeated or
field.cpp_type != FieldDescriptor.CPPTYPE_MESSAGE):
raise ValueError('Error: Field {0} in message {1} is not a singular '
'message field and cannot have sub-fields.'.format(
name, source_descriptor.full_name))
if source.HasField(name):
_MergeMessage(
child, getattr(source, name), getattr(destination, name),
replace_message, replace_repeated)
continue
if field.is_repeated:
if replace_repeated:
destination.ClearField(_StrConvert(name))
repeated_source = getattr(source, name)
repeated_destination = getattr(destination, name)
repeated_destination.MergeFrom(repeated_source)
else:
if field.cpp_type == FieldDescriptor.CPPTYPE_MESSAGE:
if replace_message:
destination.ClearField(_StrConvert(name))
if source.HasField(name):
getattr(destination, name).MergeFrom(getattr(source, name))
elif not field.has_presence or source.HasField(name):
setattr(destination, name, getattr(source, name))
else:
destination.ClearField(_StrConvert(name))
def _AddFieldPaths(node, prefix, field_mask):
"""Adds the field paths descended from node to field_mask."""
if not node and prefix:
field_mask.paths.append(prefix)
return
for name in sorted(node):
if prefix:
child_path = prefix + '.' + name
else:
child_path = name
_AddFieldPaths(node[name], child_path, field_mask)
@@ -0,0 +1,55 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Defines a listener interface for observing certain
state transitions on Message objects.
Also defines a null implementation of this interface.
"""
__author__ = 'robinson@google.com (Will Robinson)'
class MessageListener(object):
"""Listens for modifications made to a message. Meant to be registered via
Message._SetListener().
Attributes:
dirty: If True, then calling Modified() would be a no-op. This can be
used to avoid these calls entirely in the common case.
"""
def Modified(self):
"""Called every time the message is modified in such a way that the parent
message may need to be updated. This currently means either:
(a) The message was modified for the first time, so the parent message
should henceforth mark the message as present.
(b) The message's cached byte size became dirty -- i.e. the message was
modified for the first time after a previous call to ByteSize().
Therefore the parent should also mark its byte size as dirty.
Note that (a) implies (b), since new objects start out with a client cached
size (zero). However, we document (a) explicitly because it is important.
Modified() will *only* be called in response to one of these two events --
not every time the sub-message is modified.
Note that if the listener's |dirty| attribute is true, then calling
Modified at the moment would be a no-op, so it can be skipped. Performance-
sensitive callers should check this attribute directly before calling since
it will be true most of the time.
"""
raise NotImplementedError
class NullMessageListener(object):
"""No-op MessageListener implementation."""
def Modified(self):
pass
@@ -0,0 +1,5 @@
"""
This file contains the serialized FeatureSetDefaults object corresponding to
the Pure Python runtime. This is used for feature resolution under Editions.
"""
_PROTOBUF_INTERNAL_PYTHON_EDITION_DEFAULTS = b"\n\027\030\204\007\"\000*\020\010\001\020\002\030\002 \003(\0010\0028\002@\001\n\027\030\347\007\"\000*\020\010\002\020\001\030\001 \002(\0010\0018\002@\001\n\027\030\350\007\"\014\010\001\020\001\030\001 \002(\0010\001*\0048\002@\001\n\027\030\351\007\"\020\010\001\020\001\030\001 \002(\0010\0018\001@\002*\000 \346\007(\351\007"
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,128 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""A subclass of unittest.TestCase which checks for reference leaks.
To use:
- Use testing_refleak.BaseTestCase instead of unittest.TestCase
- Configure and compile Python with --with-pydebug
If sys.gettotalrefcount() is not available (because Python was built without
the Py_DEBUG option), then this module is a no-op and tests will run normally.
"""
import copyreg
import gc
import sys
import unittest
class LocalTestResult(unittest.TestResult):
"""A TestResult which forwards events to a parent object, except for Skips."""
def __init__(self, parent_result):
unittest.TestResult.__init__(self)
self.parent_result = parent_result
def addError(self, test, error):
self.parent_result.addError(test, error)
def addFailure(self, test, error):
self.parent_result.addFailure(test, error)
def addSkip(self, test, reason):
pass
def addDuration(self, test, duration):
pass
class ReferenceLeakCheckerMixin(object):
"""A mixin class for TestCase, which checks reference counts."""
NB_RUNS = 3
def run(self, result=None):
testMethod = getattr(self, self._testMethodName)
expecting_failure_method = getattr(testMethod, "__unittest_expecting_failure__", False)
expecting_failure_class = getattr(self, "__unittest_expecting_failure__", False)
if expecting_failure_class or expecting_failure_method:
return
# python_message.py registers all Message classes to some pickle global
# registry, which makes the classes immortal.
# We save a copy of this registry, and reset it before we could references.
self._saved_pickle_registry = copyreg.dispatch_table.copy()
# Run the test twice, to warm up the instance attributes.
super(ReferenceLeakCheckerMixin, self).run(result=result)
super(ReferenceLeakCheckerMixin, self).run(result=result)
local_result = LocalTestResult(result)
num_flakes = 0
refcount_deltas = []
# Observe the refcount, then create oldrefcount which actually makes the
# refcount 1 higher than the recorded value immediately
oldrefcount = self._getRefcounts()
while len(refcount_deltas) < self.NB_RUNS:
oldrefcount = self._getRefcounts()
super(ReferenceLeakCheckerMixin, self).run(result=local_result)
newrefcount = self._getRefcounts()
# If the GC was able to collect some objects after the call to run() that
# it could not collect before the call, then the counts won't match.
if newrefcount < oldrefcount and num_flakes < 2:
# This result is (probably) a flake -- garbage collectors aren't very
# predictable, but a lower ending refcount is the opposite of the
# failure we are testing for. If the result is repeatable, then we will
# eventually report it, but not after trying to eliminate it.
num_flakes += 1
continue
num_flakes = 0
refcount_deltas.append(newrefcount - oldrefcount)
print(refcount_deltas, self)
try:
self.assertEqual(refcount_deltas, [0] * self.NB_RUNS)
except Exception: # pylint: disable=broad-except
result.addError(self, sys.exc_info())
def _getRefcounts(self):
if hasattr(sys, "_clear_internal_caches"): # Since 3.13
sys._clear_internal_caches() # pylint: disable=protected-access
else:
sys._clear_type_cache() # pylint: disable=protected-access
copyreg.dispatch_table.clear()
copyreg.dispatch_table.update(self._saved_pickle_registry)
# It is sometimes necessary to gc.collect() multiple times, to ensure
# that all objects can be collected.
gc.collect()
gc.collect()
gc.collect()
return sys.gettotalrefcount()
if hasattr(sys, 'gettotalrefcount'):
def TestCase(test_class):
new_bases = (ReferenceLeakCheckerMixin,) + test_class.__bases__
new_class = type(test_class)(
test_class.__name__, new_bases, dict(test_class.__dict__))
return new_class
SkipReferenceLeakChecker = unittest.skip
else:
# When PyDEBUG is not enabled, run the tests normally.
def TestCase(test_class):
return test_class
def SkipReferenceLeakChecker(reason):
del reason # Don't skip, so don't need a reason.
def Same(func):
return func
return Same
@@ -0,0 +1,455 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Provides type checking routines.
This module defines type checking utilities in the forms of dictionaries:
VALUE_CHECKERS: A dictionary of field types and a value validation object.
TYPE_TO_BYTE_SIZE_FN: A dictionary with field types and a size computing
function.
TYPE_TO_SERIALIZE_METHOD: A dictionary with field types and serialization
function.
FIELD_TYPE_TO_WIRE_TYPE: A dictionary with field typed and their
corresponding wire types.
TYPE_TO_DESERIALIZE_METHOD: A dictionary with field types and deserialization
function.
"""
__author__ = 'robinson@google.com (Will Robinson)'
import numbers
import struct
import warnings
from google.protobuf import descriptor
from google.protobuf.internal import decoder
from google.protobuf.internal import encoder
from google.protobuf.internal import wire_format
_FieldDescriptor = descriptor.FieldDescriptor
# TODO: Remove this warning count after 34.0
# Assign bool to int/enum warnings will print 100 times at most which should
# be enough for users to notice and do not cause timeout.
_BoolWarningCount = 100
def TruncateToFourByteFloat(original):
return struct.unpack('<f', struct.pack('<f', original))[0]
def ToShortestFloat(original):
"""Returns the shortest float that has same value in wire."""
# All 4 byte floats have between 6 and 9 significant digits, so we
# start with 6 as the lower bound.
# It has to be iterative because use '.9g' directly can not get rid
# of the noises for most values. For example if set a float_field=0.9
# use '.9g' will print 0.899999976.
precision = 6
rounded = float('{0:.{1}g}'.format(original, precision))
while TruncateToFourByteFloat(rounded) != original:
precision += 1
rounded = float('{0:.{1}g}'.format(original, precision))
return rounded
def GetTypeChecker(field):
"""Returns a type checker for a message field of the specified types.
Args:
field: FieldDescriptor object for this field.
Returns:
An instance of TypeChecker which can be used to verify the types
of values assigned to a field of the specified type.
"""
if (field.cpp_type == _FieldDescriptor.CPPTYPE_STRING and
field.type == _FieldDescriptor.TYPE_STRING):
return UnicodeValueChecker()
if field.cpp_type == _FieldDescriptor.CPPTYPE_ENUM:
if field.enum_type.is_closed:
return EnumValueChecker(field.enum_type)
else:
# When open enums are supported, any int32 can be assigned.
return _VALUE_CHECKERS[_FieldDescriptor.CPPTYPE_INT32]
return _VALUE_CHECKERS[field.cpp_type]
# None of the typecheckers below make any attempt to guard against people
# subclassing builtin types and doing weird things. We're not trying to
# protect against malicious clients here, just people accidentally shooting
# themselves in the foot in obvious ways.
class TypeChecker(object):
"""Type checker used to catch type errors as early as possible
when the client is setting scalar fields in protocol messages.
"""
def __init__(self, *acceptable_types):
self._acceptable_types = acceptable_types
def CheckValue(self, proposed_value):
"""Type check the provided value and return it.
The returned value might have been normalized to another type.
"""
if not isinstance(proposed_value, self._acceptable_types):
message = ('%.1024r has type %s, but expected one of: %s' %
(proposed_value, type(proposed_value), self._acceptable_types))
raise TypeError(message)
return proposed_value
class TypeCheckerWithDefault(TypeChecker):
def __init__(self, default_value, *acceptable_types):
TypeChecker.__init__(self, *acceptable_types)
self._default_value = default_value
def DefaultValue(self):
return self._default_value
class BoolValueChecker(object):
"""Type checker used for bool fields."""
def CheckValue(self, proposed_value):
if not hasattr(proposed_value, '__index__'):
# Under NumPy 2.3, numpy.bool does not have an __index__ method.
if (type(proposed_value).__module__ == 'numpy' and
type(proposed_value).__name__ == 'bool'):
return bool(proposed_value)
message = ('%.1024r has type %s, but expected one of: %s' %
(proposed_value, type(proposed_value), (bool, int)))
raise TypeError(message)
if (type(proposed_value).__module__ == 'numpy' and
type(proposed_value).__name__ == 'ndarray'):
message = ('%.1024r has type %s, but expected one of: %s' %
(proposed_value, type(proposed_value), (bool, int)))
raise TypeError(message)
return bool(proposed_value)
def DefaultValue(self):
return False
# IntValueChecker and its subclasses perform integer type-checks
# and bounds-checks.
class IntValueChecker(object):
"""Checker used for integer fields. Performs type-check and range check."""
def CheckValue(self, proposed_value):
global _BoolWarningCount
if type(proposed_value) == bool and _BoolWarningCount > 0:
_BoolWarningCount -= 1
message = (
'%.1024r has type %s, but expected one of: %s. This warning '
'will turn into error in 7.34.0, please fix it before that.'
% (
proposed_value,
type(proposed_value),
(int,),
)
)
# TODO: Raise errors in 2026 Q1 release
warnings.warn(message)
if not hasattr(proposed_value, '__index__') or (
type(proposed_value).__module__ == 'numpy' and
type(proposed_value).__name__ == 'ndarray'):
message = ('%.1024r has type %s, but expected one of: %s' %
(proposed_value, type(proposed_value), (int,)))
raise TypeError(message)
if not self._MIN <= int(proposed_value) <= self._MAX:
raise ValueError('Value out of range: %d' % proposed_value)
# We force all values to int to make alternate implementations where the
# distinction is more significant (e.g. the C++ implementation) simpler.
proposed_value = int(proposed_value)
return proposed_value
def DefaultValue(self):
return 0
class EnumValueChecker(object):
"""Checker used for enum fields. Performs type-check and range check."""
def __init__(self, enum_type):
self._enum_type = enum_type
def CheckValue(self, proposed_value):
global _BoolWarningCount
if type(proposed_value) == bool and _BoolWarningCount > 0:
_BoolWarningCount -= 1
message = (
'%.1024r has type %s, but expected one of: %s. This warning '
'will turn into error in 7.34.0, please fix it before that.'
% (
proposed_value,
type(proposed_value),
(int,),
)
)
# TODO: Raise errors in 2026 Q1 release
warnings.warn(message)
if not isinstance(proposed_value, numbers.Integral):
message = ('%.1024r has type %s, but expected one of: %s' %
(proposed_value, type(proposed_value), (int,)))
raise TypeError(message)
if int(proposed_value) not in self._enum_type.values_by_number:
raise ValueError('Unknown enum value: %d' % proposed_value)
return proposed_value
def DefaultValue(self):
return self._enum_type.values[0].number
class UnicodeValueChecker(object):
"""Checker used for string fields.
Always returns a unicode value, even if the input is of type str.
"""
def CheckValue(self, proposed_value):
if not isinstance(proposed_value, (bytes, str)):
message = ('%.1024r has type %s, but expected one of: %s' %
(proposed_value, type(proposed_value), (bytes, str)))
raise TypeError(message)
# If the value is of type 'bytes' make sure that it is valid UTF-8 data.
if isinstance(proposed_value, bytes):
try:
proposed_value = proposed_value.decode('utf-8')
except UnicodeDecodeError:
raise ValueError('%.1024r has type bytes, but isn\'t valid UTF-8 '
'encoding. Non-UTF-8 strings must be converted to '
'unicode objects before being added.' %
(proposed_value))
else:
try:
proposed_value.encode('utf8')
except UnicodeEncodeError:
raise ValueError('%.1024r isn\'t a valid unicode string and '
'can\'t be encoded in UTF-8.'%
(proposed_value))
return proposed_value
def DefaultValue(self):
return u""
class Int32ValueChecker(IntValueChecker):
# We're sure to use ints instead of longs here since comparison may be more
# efficient.
_MIN = -2147483648
_MAX = 2147483647
class Uint32ValueChecker(IntValueChecker):
_MIN = 0
_MAX = (1 << 32) - 1
class Int64ValueChecker(IntValueChecker):
_MIN = -(1 << 63)
_MAX = (1 << 63) - 1
class Uint64ValueChecker(IntValueChecker):
_MIN = 0
_MAX = (1 << 64) - 1
# The max 4 bytes float is about 3.4028234663852886e+38
_FLOAT_MAX = float.fromhex('0x1.fffffep+127')
_FLOAT_MIN = -_FLOAT_MAX
_MAX_FLOAT_AS_DOUBLE_ROUNDED = 3.4028235677973366e38
_INF = float('inf')
_NEG_INF = float('-inf')
class DoubleValueChecker(object):
"""Checker used for double fields.
Performs type-check and range check.
"""
def CheckValue(self, proposed_value):
"""Check and convert proposed_value to float."""
if (not hasattr(proposed_value, '__float__') and
not hasattr(proposed_value, '__index__')) or (
type(proposed_value).__module__ == 'numpy' and
type(proposed_value).__name__ == 'ndarray'):
message = ('%.1024r has type %s, but expected one of: int, float' %
(proposed_value, type(proposed_value)))
raise TypeError(message)
return float(proposed_value)
def DefaultValue(self):
return 0.0
class FloatValueChecker(DoubleValueChecker):
"""Checker used for float fields.
Performs type-check and range check.
Values exceeding a 32-bit float will be converted to inf/-inf.
"""
def CheckValue(self, proposed_value):
"""Check and convert proposed_value to float."""
converted_value = super().CheckValue(proposed_value)
# This inf rounding matches the C++ proto SafeDoubleToFloat logic.
if converted_value > _FLOAT_MAX:
if converted_value <= _MAX_FLOAT_AS_DOUBLE_ROUNDED:
return _FLOAT_MAX
return _INF
if converted_value < _FLOAT_MIN:
if converted_value >= -_MAX_FLOAT_AS_DOUBLE_ROUNDED:
return _FLOAT_MIN
return _NEG_INF
return TruncateToFourByteFloat(converted_value)
# Type-checkers for all scalar CPPTYPEs.
_VALUE_CHECKERS = {
_FieldDescriptor.CPPTYPE_INT32: Int32ValueChecker(),
_FieldDescriptor.CPPTYPE_INT64: Int64ValueChecker(),
_FieldDescriptor.CPPTYPE_UINT32: Uint32ValueChecker(),
_FieldDescriptor.CPPTYPE_UINT64: Uint64ValueChecker(),
_FieldDescriptor.CPPTYPE_DOUBLE: DoubleValueChecker(),
_FieldDescriptor.CPPTYPE_FLOAT: FloatValueChecker(),
_FieldDescriptor.CPPTYPE_BOOL: BoolValueChecker(),
_FieldDescriptor.CPPTYPE_STRING: TypeCheckerWithDefault(b'', bytes),
}
# Map from field type to a function F, such that F(field_num, value)
# gives the total byte size for a value of the given type. This
# byte size includes tag information and any other additional space
# associated with serializing "value".
TYPE_TO_BYTE_SIZE_FN = {
_FieldDescriptor.TYPE_DOUBLE: wire_format.DoubleByteSize,
_FieldDescriptor.TYPE_FLOAT: wire_format.FloatByteSize,
_FieldDescriptor.TYPE_INT64: wire_format.Int64ByteSize,
_FieldDescriptor.TYPE_UINT64: wire_format.UInt64ByteSize,
_FieldDescriptor.TYPE_INT32: wire_format.Int32ByteSize,
_FieldDescriptor.TYPE_FIXED64: wire_format.Fixed64ByteSize,
_FieldDescriptor.TYPE_FIXED32: wire_format.Fixed32ByteSize,
_FieldDescriptor.TYPE_BOOL: wire_format.BoolByteSize,
_FieldDescriptor.TYPE_STRING: wire_format.StringByteSize,
_FieldDescriptor.TYPE_GROUP: wire_format.GroupByteSize,
_FieldDescriptor.TYPE_MESSAGE: wire_format.MessageByteSize,
_FieldDescriptor.TYPE_BYTES: wire_format.BytesByteSize,
_FieldDescriptor.TYPE_UINT32: wire_format.UInt32ByteSize,
_FieldDescriptor.TYPE_ENUM: wire_format.EnumByteSize,
_FieldDescriptor.TYPE_SFIXED32: wire_format.SFixed32ByteSize,
_FieldDescriptor.TYPE_SFIXED64: wire_format.SFixed64ByteSize,
_FieldDescriptor.TYPE_SINT32: wire_format.SInt32ByteSize,
_FieldDescriptor.TYPE_SINT64: wire_format.SInt64ByteSize
}
# Maps from field types to encoder constructors.
TYPE_TO_ENCODER = {
_FieldDescriptor.TYPE_DOUBLE: encoder.DoubleEncoder,
_FieldDescriptor.TYPE_FLOAT: encoder.FloatEncoder,
_FieldDescriptor.TYPE_INT64: encoder.Int64Encoder,
_FieldDescriptor.TYPE_UINT64: encoder.UInt64Encoder,
_FieldDescriptor.TYPE_INT32: encoder.Int32Encoder,
_FieldDescriptor.TYPE_FIXED64: encoder.Fixed64Encoder,
_FieldDescriptor.TYPE_FIXED32: encoder.Fixed32Encoder,
_FieldDescriptor.TYPE_BOOL: encoder.BoolEncoder,
_FieldDescriptor.TYPE_STRING: encoder.StringEncoder,
_FieldDescriptor.TYPE_GROUP: encoder.GroupEncoder,
_FieldDescriptor.TYPE_MESSAGE: encoder.MessageEncoder,
_FieldDescriptor.TYPE_BYTES: encoder.BytesEncoder,
_FieldDescriptor.TYPE_UINT32: encoder.UInt32Encoder,
_FieldDescriptor.TYPE_ENUM: encoder.EnumEncoder,
_FieldDescriptor.TYPE_SFIXED32: encoder.SFixed32Encoder,
_FieldDescriptor.TYPE_SFIXED64: encoder.SFixed64Encoder,
_FieldDescriptor.TYPE_SINT32: encoder.SInt32Encoder,
_FieldDescriptor.TYPE_SINT64: encoder.SInt64Encoder,
}
# Maps from field types to sizer constructors.
TYPE_TO_SIZER = {
_FieldDescriptor.TYPE_DOUBLE: encoder.DoubleSizer,
_FieldDescriptor.TYPE_FLOAT: encoder.FloatSizer,
_FieldDescriptor.TYPE_INT64: encoder.Int64Sizer,
_FieldDescriptor.TYPE_UINT64: encoder.UInt64Sizer,
_FieldDescriptor.TYPE_INT32: encoder.Int32Sizer,
_FieldDescriptor.TYPE_FIXED64: encoder.Fixed64Sizer,
_FieldDescriptor.TYPE_FIXED32: encoder.Fixed32Sizer,
_FieldDescriptor.TYPE_BOOL: encoder.BoolSizer,
_FieldDescriptor.TYPE_STRING: encoder.StringSizer,
_FieldDescriptor.TYPE_GROUP: encoder.GroupSizer,
_FieldDescriptor.TYPE_MESSAGE: encoder.MessageSizer,
_FieldDescriptor.TYPE_BYTES: encoder.BytesSizer,
_FieldDescriptor.TYPE_UINT32: encoder.UInt32Sizer,
_FieldDescriptor.TYPE_ENUM: encoder.EnumSizer,
_FieldDescriptor.TYPE_SFIXED32: encoder.SFixed32Sizer,
_FieldDescriptor.TYPE_SFIXED64: encoder.SFixed64Sizer,
_FieldDescriptor.TYPE_SINT32: encoder.SInt32Sizer,
_FieldDescriptor.TYPE_SINT64: encoder.SInt64Sizer,
}
# Maps from field type to a decoder constructor.
TYPE_TO_DECODER = {
_FieldDescriptor.TYPE_DOUBLE: decoder.DoubleDecoder,
_FieldDescriptor.TYPE_FLOAT: decoder.FloatDecoder,
_FieldDescriptor.TYPE_INT64: decoder.Int64Decoder,
_FieldDescriptor.TYPE_UINT64: decoder.UInt64Decoder,
_FieldDescriptor.TYPE_INT32: decoder.Int32Decoder,
_FieldDescriptor.TYPE_FIXED64: decoder.Fixed64Decoder,
_FieldDescriptor.TYPE_FIXED32: decoder.Fixed32Decoder,
_FieldDescriptor.TYPE_BOOL: decoder.BoolDecoder,
_FieldDescriptor.TYPE_STRING: decoder.StringDecoder,
_FieldDescriptor.TYPE_GROUP: decoder.GroupDecoder,
_FieldDescriptor.TYPE_MESSAGE: decoder.MessageDecoder,
_FieldDescriptor.TYPE_BYTES: decoder.BytesDecoder,
_FieldDescriptor.TYPE_UINT32: decoder.UInt32Decoder,
_FieldDescriptor.TYPE_ENUM: decoder.EnumDecoder,
_FieldDescriptor.TYPE_SFIXED32: decoder.SFixed32Decoder,
_FieldDescriptor.TYPE_SFIXED64: decoder.SFixed64Decoder,
_FieldDescriptor.TYPE_SINT32: decoder.SInt32Decoder,
_FieldDescriptor.TYPE_SINT64: decoder.SInt64Decoder,
}
# Maps from field type to expected wiretype.
FIELD_TYPE_TO_WIRE_TYPE = {
_FieldDescriptor.TYPE_DOUBLE: wire_format.WIRETYPE_FIXED64,
_FieldDescriptor.TYPE_FLOAT: wire_format.WIRETYPE_FIXED32,
_FieldDescriptor.TYPE_INT64: wire_format.WIRETYPE_VARINT,
_FieldDescriptor.TYPE_UINT64: wire_format.WIRETYPE_VARINT,
_FieldDescriptor.TYPE_INT32: wire_format.WIRETYPE_VARINT,
_FieldDescriptor.TYPE_FIXED64: wire_format.WIRETYPE_FIXED64,
_FieldDescriptor.TYPE_FIXED32: wire_format.WIRETYPE_FIXED32,
_FieldDescriptor.TYPE_BOOL: wire_format.WIRETYPE_VARINT,
_FieldDescriptor.TYPE_STRING:
wire_format.WIRETYPE_LENGTH_DELIMITED,
_FieldDescriptor.TYPE_GROUP: wire_format.WIRETYPE_START_GROUP,
_FieldDescriptor.TYPE_MESSAGE:
wire_format.WIRETYPE_LENGTH_DELIMITED,
_FieldDescriptor.TYPE_BYTES:
wire_format.WIRETYPE_LENGTH_DELIMITED,
_FieldDescriptor.TYPE_UINT32: wire_format.WIRETYPE_VARINT,
_FieldDescriptor.TYPE_ENUM: wire_format.WIRETYPE_VARINT,
_FieldDescriptor.TYPE_SFIXED32: wire_format.WIRETYPE_FIXED32,
_FieldDescriptor.TYPE_SFIXED64: wire_format.WIRETYPE_FIXED64,
_FieldDescriptor.TYPE_SINT32: wire_format.WIRETYPE_VARINT,
_FieldDescriptor.TYPE_SINT64: wire_format.WIRETYPE_VARINT,
}
@@ -0,0 +1,695 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Contains well known classes.
This files defines well known classes which need extra maintenance including:
- Any
- Duration
- FieldMask
- Struct
- Timestamp
"""
__author__ = 'jieluo@google.com (Jie Luo)'
import calendar
import collections.abc
import datetime
from typing import Union
import warnings
from google.protobuf.internal import field_mask
FieldMask = field_mask.FieldMask
_TIMESTAMPFORMAT = '%Y-%m-%dT%H:%M:%S'
_NANOS_PER_SECOND = 1000000000
_NANOS_PER_MILLISECOND = 1000000
_NANOS_PER_MICROSECOND = 1000
_MILLIS_PER_SECOND = 1000
_MICROS_PER_SECOND = 1000000
_SECONDS_PER_DAY = 24 * 3600
_DURATION_SECONDS_MAX = 315576000000
_TIMESTAMP_SECONDS_MIN = -62135596800
_TIMESTAMP_SECONDS_MAX = 253402300799
_EPOCH_DATETIME_NAIVE = datetime.datetime(1970, 1, 1, tzinfo=None)
_EPOCH_DATETIME_AWARE = _EPOCH_DATETIME_NAIVE.replace(
tzinfo=datetime.timezone.utc
)
class Any(object):
"""Class for Any Message type."""
__slots__ = ()
def Pack(
self, msg, type_url_prefix='type.googleapis.com/', deterministic=None
):
"""Packs the specified message into current Any message."""
if len(type_url_prefix) < 1 or type_url_prefix[-1] != '/':
self.type_url = '%s/%s' % (type_url_prefix, msg.DESCRIPTOR.full_name)
else:
self.type_url = '%s%s' % (type_url_prefix, msg.DESCRIPTOR.full_name)
self.value = msg.SerializeToString(deterministic=deterministic)
def Unpack(self, msg):
"""Unpacks the current Any message into specified message."""
descriptor = msg.DESCRIPTOR
if not self.Is(descriptor):
return False
msg.ParseFromString(self.value)
return True
def TypeName(self):
"""Returns the protobuf type name of the inner message."""
# Only last part is to be used: b/25630112
return self.type_url.rpartition('/')[2]
def Is(self, descriptor):
"""Checks if this Any represents the given protobuf type."""
return '/' in self.type_url and self.TypeName() == descriptor.full_name
class Timestamp(object):
"""Class for Timestamp message type."""
__slots__ = ()
def ToJsonString(self):
"""Converts Timestamp to RFC 3339 date string format.
Returns:
A string converted from timestamp. The string is always Z-normalized
and uses 3, 6 or 9 fractional digits as required to represent the
exact time. Example of the return format: '1972-01-01T10:00:20.021Z'
"""
_CheckTimestampValid(self.seconds, self.nanos)
nanos = self.nanos
seconds = self.seconds % _SECONDS_PER_DAY
days = (self.seconds - seconds) // _SECONDS_PER_DAY
dt = datetime.datetime(1970, 1, 1) + datetime.timedelta(days, seconds)
result = dt.isoformat()
if (nanos % 1e9) == 0:
# If there are 0 fractional digits, the fractional
# point '.' should be omitted when serializing.
return result + 'Z'
if (nanos % 1e6) == 0:
# Serialize 3 fractional digits.
return result + '.%03dZ' % (nanos / 1e6)
if (nanos % 1e3) == 0:
# Serialize 6 fractional digits.
return result + '.%06dZ' % (nanos / 1e3)
# Serialize 9 fractional digits.
return result + '.%09dZ' % nanos
def FromJsonString(self, value):
"""Parse a RFC 3339 date string format to Timestamp.
Args:
value: A date string. Any fractional digits (or none) and any offset are
accepted as long as they fit into nano-seconds precision. Example of
accepted format: '1972-01-01T10:00:20.021-05:00'
Raises:
ValueError: On parsing problems.
"""
if not isinstance(value, str):
raise ValueError('Timestamp JSON value not a string: {!r}'.format(value))
timezone_offset = value.find('Z')
if timezone_offset == -1:
timezone_offset = value.find('+')
if timezone_offset == -1:
timezone_offset = value.rfind('-')
if timezone_offset == -1:
raise ValueError(
'Failed to parse timestamp: missing valid timezone offset.'
)
time_value = value[0:timezone_offset]
# Parse datetime and nanos.
point_position = time_value.find('.')
if point_position == -1:
second_value = time_value
nano_value = ''
else:
second_value = time_value[:point_position]
nano_value = time_value[point_position + 1 :]
if 't' in second_value:
raise ValueError(
"time data '{0}' does not match format '%Y-%m-%dT%H:%M:%S', "
"lowercase 't' is not accepted".format(second_value)
)
date_object = datetime.datetime.strptime(second_value, _TIMESTAMPFORMAT)
td = date_object - datetime.datetime(1970, 1, 1)
seconds = td.seconds + td.days * _SECONDS_PER_DAY
if len(nano_value) > 9:
raise ValueError(
'Failed to parse Timestamp: nanos {0} more than '
'9 fractional digits.'.format(nano_value)
)
if nano_value:
nanos = round(float('0.' + nano_value) * 1e9)
else:
nanos = 0
# Parse timezone offsets.
if value[timezone_offset] == 'Z':
if len(value) != timezone_offset + 1:
raise ValueError(
'Failed to parse timestamp: invalid trailing data {0}.'.format(
value
)
)
else:
timezone = value[timezone_offset:]
pos = timezone.find(':')
if pos == -1:
raise ValueError('Invalid timezone offset value: {0}.'.format(timezone))
if timezone[0] == '+':
seconds -= (int(timezone[1:pos]) * 60 + int(timezone[pos + 1 :])) * 60
else:
seconds += (int(timezone[1:pos]) * 60 + int(timezone[pos + 1 :])) * 60
# Set seconds and nanos
_CheckTimestampValid(seconds, nanos)
self.seconds = int(seconds)
self.nanos = int(nanos)
def GetCurrentTime(self):
"""Get the current UTC into Timestamp."""
self.FromDatetime(datetime.datetime.now(tz=datetime.timezone.utc))
def ToNanoseconds(self):
"""Converts Timestamp to nanoseconds since epoch."""
_CheckTimestampValid(self.seconds, self.nanos)
return self.seconds * _NANOS_PER_SECOND + self.nanos
def ToMicroseconds(self):
"""Converts Timestamp to microseconds since epoch."""
_CheckTimestampValid(self.seconds, self.nanos)
return (
self.seconds * _MICROS_PER_SECOND + self.nanos // _NANOS_PER_MICROSECOND
)
def ToMilliseconds(self):
"""Converts Timestamp to milliseconds since epoch."""
_CheckTimestampValid(self.seconds, self.nanos)
return (
self.seconds * _MILLIS_PER_SECOND + self.nanos // _NANOS_PER_MILLISECOND
)
def ToSeconds(self):
"""Converts Timestamp to seconds since epoch."""
_CheckTimestampValid(self.seconds, self.nanos)
return self.seconds
def FromNanoseconds(self, nanos):
"""Converts nanoseconds since epoch to Timestamp."""
seconds = nanos // _NANOS_PER_SECOND
nanos = nanos % _NANOS_PER_SECOND
_CheckTimestampValid(seconds, nanos)
self.seconds = seconds
self.nanos = nanos
def FromMicroseconds(self, micros):
"""Converts microseconds since epoch to Timestamp."""
seconds = micros // _MICROS_PER_SECOND
nanos = (micros % _MICROS_PER_SECOND) * _NANOS_PER_MICROSECOND
_CheckTimestampValid(seconds, nanos)
self.seconds = seconds
self.nanos = nanos
def FromMilliseconds(self, millis):
"""Converts milliseconds since epoch to Timestamp."""
seconds = millis // _MILLIS_PER_SECOND
nanos = (millis % _MILLIS_PER_SECOND) * _NANOS_PER_MILLISECOND
_CheckTimestampValid(seconds, nanos)
self.seconds = seconds
self.nanos = nanos
def FromSeconds(self, seconds):
"""Converts seconds since epoch to Timestamp."""
_CheckTimestampValid(seconds, 0)
self.seconds = seconds
self.nanos = 0
def ToDatetime(self, tzinfo=None):
"""Converts Timestamp to a datetime.
Args:
tzinfo: A datetime.tzinfo subclass; defaults to None.
Returns:
If tzinfo is None, returns a timezone-naive UTC datetime (with no timezone
information, i.e. not aware that it's UTC).
Otherwise, returns a timezone-aware datetime in the input timezone.
"""
# Using datetime.fromtimestamp for this would avoid constructing an extra
# timedelta object and possibly an extra datetime. Unfortunately, that has
# the disadvantage of not handling the full precision (on all platforms, see
# https://github.com/python/cpython/issues/109849) or full range (on some
# platforms, see https://github.com/python/cpython/issues/110042) of
# datetime.
_CheckTimestampValid(self.seconds, self.nanos)
delta = datetime.timedelta(
seconds=self.seconds,
microseconds=_RoundTowardZero(self.nanos, _NANOS_PER_MICROSECOND),
)
if tzinfo is None:
return _EPOCH_DATETIME_NAIVE + delta
else:
# Note the tz conversion has to come after the timedelta arithmetic.
return (_EPOCH_DATETIME_AWARE + delta).astimezone(tzinfo)
def FromDatetime(self, dt):
"""Converts datetime to Timestamp.
Args:
dt: A datetime. If it's timezone-naive, it's assumed to be in UTC.
"""
# Using this guide: http://wiki.python.org/moin/WorkingWithTime
# And this conversion guide: http://docs.python.org/library/time.html
# Turn the date parameter into a tuple (struct_time) that can then be
# manipulated into a long value of seconds. During the conversion from
# struct_time to long, the source date in UTC, and so it follows that the
# correct transformation is calendar.timegm()
try:
seconds = calendar.timegm(dt.utctimetuple())
nanos = dt.microsecond * _NANOS_PER_MICROSECOND
except AttributeError as e:
raise AttributeError(
'Fail to convert to Timestamp. Expected a datetime like '
'object got {0} : {1}'.format(type(dt).__name__, e)
) from e
_CheckTimestampValid(seconds, nanos)
self.seconds = seconds
self.nanos = nanos
def _internal_assign(self, dt):
self.FromDatetime(dt)
def __add__(self, value) -> datetime.datetime:
if isinstance(value, Duration):
return self.ToDatetime() + value.ToTimedelta()
return self.ToDatetime() + value
__radd__ = __add__
def __sub__(self, value) -> Union[datetime.datetime, datetime.timedelta]:
if isinstance(value, Timestamp):
return self.ToDatetime() - value.ToDatetime()
elif isinstance(value, Duration):
return self.ToDatetime() - value.ToTimedelta()
return self.ToDatetime() - value
def __rsub__(self, dt) -> datetime.timedelta:
return dt - self.ToDatetime()
def _CheckTimestampValid(seconds, nanos):
if seconds < _TIMESTAMP_SECONDS_MIN or seconds > _TIMESTAMP_SECONDS_MAX:
raise ValueError(
'Timestamp is not valid: Seconds {0} must be in range '
'[-62135596800, 253402300799].'.format(seconds))
if nanos < 0 or nanos >= _NANOS_PER_SECOND:
raise ValueError(
'Timestamp is not valid: Nanos {} must be in a range '
'[0, 999999].'.format(nanos)
)
class Duration(object):
"""Class for Duration message type."""
__slots__ = ()
def ToJsonString(self):
"""Converts Duration to string format.
Returns:
A string converted from self. The string format will contains
3, 6, or 9 fractional digits depending on the precision required to
represent the exact Duration value. For example: "1s", "1.010s",
"1.000000100s", "-3.100s"
"""
_CheckDurationValid(self.seconds, self.nanos)
if self.seconds < 0 or self.nanos < 0:
result = '-'
seconds = -self.seconds + int((0 - self.nanos) // 1e9)
nanos = (0 - self.nanos) % 1e9
else:
result = ''
seconds = self.seconds + int(self.nanos // 1e9)
nanos = self.nanos % 1e9
result += '%d' % seconds
if (nanos % 1e9) == 0:
# If there are 0 fractional digits, the fractional
# point '.' should be omitted when serializing.
return result + 's'
if (nanos % 1e6) == 0:
# Serialize 3 fractional digits.
return result + '.%03ds' % (nanos / 1e6)
if (nanos % 1e3) == 0:
# Serialize 6 fractional digits.
return result + '.%06ds' % (nanos / 1e3)
# Serialize 9 fractional digits.
return result + '.%09ds' % nanos
def FromJsonString(self, value):
"""Converts a string to Duration.
Args:
value: A string to be converted. The string must end with 's'. Any
fractional digits (or none) are accepted as long as they fit into
precision. For example: "1s", "1.01s", "1.0000001s", "-3.100s
Raises:
ValueError: On parsing problems.
"""
if not isinstance(value, str):
raise ValueError('Duration JSON value not a string: {!r}'.format(value))
if len(value) < 1 or value[-1] != 's':
raise ValueError('Duration must end with letter "s": {0}.'.format(value))
try:
pos = value.find('.')
if pos == -1:
seconds = int(value[:-1])
nanos = 0
else:
seconds = int(value[:pos])
if value[0] == '-':
nanos = int(round(float('-0{0}'.format(value[pos:-1])) * 1e9))
else:
nanos = int(round(float('0{0}'.format(value[pos:-1])) * 1e9))
_CheckDurationValid(seconds, nanos)
self.seconds = seconds
self.nanos = nanos
except ValueError as e:
raise ValueError("Couldn't parse duration: {0} : {1}.".format(value, e))
def ToNanoseconds(self):
"""Converts a Duration to nanoseconds."""
return self.seconds * _NANOS_PER_SECOND + self.nanos
def ToMicroseconds(self):
"""Converts a Duration to microseconds."""
micros = _RoundTowardZero(self.nanos, _NANOS_PER_MICROSECOND)
return self.seconds * _MICROS_PER_SECOND + micros
def ToMilliseconds(self):
"""Converts a Duration to milliseconds."""
millis = _RoundTowardZero(self.nanos, _NANOS_PER_MILLISECOND)
return self.seconds * _MILLIS_PER_SECOND + millis
def ToSeconds(self):
"""Converts a Duration to seconds."""
return self.seconds
def FromNanoseconds(self, nanos):
"""Converts nanoseconds to Duration."""
self._NormalizeDuration(
nanos // _NANOS_PER_SECOND, nanos % _NANOS_PER_SECOND
)
def FromMicroseconds(self, micros):
"""Converts microseconds to Duration."""
self._NormalizeDuration(
micros // _MICROS_PER_SECOND,
(micros % _MICROS_PER_SECOND) * _NANOS_PER_MICROSECOND,
)
def FromMilliseconds(self, millis):
"""Converts milliseconds to Duration."""
self._NormalizeDuration(
millis // _MILLIS_PER_SECOND,
(millis % _MILLIS_PER_SECOND) * _NANOS_PER_MILLISECOND,
)
def FromSeconds(self, seconds):
"""Converts seconds to Duration."""
self.seconds = seconds
self.nanos = 0
def ToTimedelta(self) -> datetime.timedelta:
"""Converts Duration to timedelta."""
return datetime.timedelta(
seconds=self.seconds,
microseconds=_RoundTowardZero(self.nanos, _NANOS_PER_MICROSECOND),
)
def FromTimedelta(self, td):
"""Converts timedelta to Duration."""
try:
self._NormalizeDuration(
td.seconds + td.days * _SECONDS_PER_DAY,
td.microseconds * _NANOS_PER_MICROSECOND,
)
except AttributeError as e:
raise AttributeError(
'Fail to convert to Duration. Expected a timedelta like '
'object got {0}: {1}'.format(type(td).__name__, e)
) from e
def _internal_assign(self, td):
self.FromTimedelta(td)
def _NormalizeDuration(self, seconds, nanos):
"""Set Duration by seconds and nanos."""
# Force nanos to be negative if the duration is negative.
if seconds < 0 and nanos > 0:
seconds += 1
nanos -= _NANOS_PER_SECOND
self.seconds = seconds
self.nanos = nanos
def __add__(self, value) -> Union[datetime.datetime, datetime.timedelta]:
if isinstance(value, Timestamp):
return self.ToTimedelta() + value.ToDatetime()
return self.ToTimedelta() + value
__radd__ = __add__
def __sub__(self, value) -> datetime.timedelta:
return self.ToTimedelta() - value
def __rsub__(self, value) -> Union[datetime.datetime, datetime.timedelta]:
return value - self.ToTimedelta()
def _CheckDurationValid(seconds, nanos):
if seconds < -_DURATION_SECONDS_MAX or seconds > _DURATION_SECONDS_MAX:
raise ValueError(
'Duration is not valid: Seconds {0} must be in range '
'[-315576000000, 315576000000].'.format(seconds)
)
if nanos <= -_NANOS_PER_SECOND or nanos >= _NANOS_PER_SECOND:
raise ValueError(
'Duration is not valid: Nanos {0} must be in range '
'[-999999999, 999999999].'.format(nanos)
)
if (nanos < 0 and seconds > 0) or (nanos > 0 and seconds < 0):
raise ValueError('Duration is not valid: Sign mismatch.')
def _RoundTowardZero(value, divider):
"""Truncates the remainder part after division."""
# For some languages, the sign of the remainder is implementation
# dependent if any of the operands is negative. Here we enforce
# "rounded toward zero" semantics. For example, for (-5) / 2 an
# implementation may give -3 as the result with the remainder being
# 1. This function ensures we always return -2 (closer to zero).
result = value // divider
remainder = value % divider
if result < 0 and remainder > 0:
return result + 1
else:
return result
def _SetStructValue(struct_value, value):
if value is None:
struct_value.null_value = 0
elif isinstance(value, bool):
# Note: this check must come before the number check because in Python
# True and False are also considered numbers.
struct_value.bool_value = value
elif isinstance(value, str):
struct_value.string_value = value
elif isinstance(value, (int, float)):
struct_value.number_value = value
elif isinstance(value, (dict, Struct)):
struct_value.struct_value.Clear()
struct_value.struct_value.update(value)
elif isinstance(value, (list, tuple, ListValue)):
struct_value.list_value.Clear()
struct_value.list_value.extend(value)
else:
raise ValueError('Unexpected type')
def _GetStructValue(struct_value):
which = struct_value.WhichOneof('kind')
if which == 'struct_value':
return struct_value.struct_value
elif which == 'null_value':
return None
elif which == 'number_value':
return struct_value.number_value
elif which == 'string_value':
return struct_value.string_value
elif which == 'bool_value':
return struct_value.bool_value
elif which == 'list_value':
return struct_value.list_value
elif which is None:
raise ValueError('Value not set')
class Struct(object):
"""Class for Struct message type."""
__slots__ = ()
def __getitem__(self, key):
return _GetStructValue(self.fields[key])
def __setitem__(self, key, value):
_SetStructValue(self.fields[key], value)
def __delitem__(self, key):
del self.fields[key]
def __len__(self):
return len(self.fields)
def __iter__(self):
return iter(self.fields)
def _internal_assign(self, dictionary):
self.Clear()
self.update(dictionary)
def _internal_compare(self, other):
size = len(self)
if size != len(other):
return False
for key, value in self.items():
if key not in other:
return False
if isinstance(other[key], (dict, list)):
if not value._internal_compare(other[key]):
return False
elif value != other[key]:
return False
return True
def keys(self): # pylint: disable=invalid-name
return self.fields.keys()
def values(self): # pylint: disable=invalid-name
return [self[key] for key in self]
def items(self): # pylint: disable=invalid-name
return [(key, self[key]) for key in self]
def get_or_create_list(self, key):
"""Returns a list for this key, creating if it didn't exist already."""
if not self.fields[key].HasField('list_value'):
# Clear will mark list_value modified which will indeed create a list.
self.fields[key].list_value.Clear()
return self.fields[key].list_value
def get_or_create_struct(self, key):
"""Returns a struct for this key, creating if it didn't exist already."""
if not self.fields[key].HasField('struct_value'):
# Clear will mark struct_value modified which will indeed create a struct.
self.fields[key].struct_value.Clear()
return self.fields[key].struct_value
def update(self, dictionary): # pylint: disable=invalid-name
for key, value in dictionary.items():
_SetStructValue(self.fields[key], value)
collections.abc.MutableMapping.register(Struct)
class ListValue(object):
"""Class for ListValue message type."""
__slots__ = ()
def __len__(self):
return len(self.values)
def append(self, value):
_SetStructValue(self.values.add(), value)
def extend(self, elem_seq):
for value in elem_seq:
self.append(value)
def __getitem__(self, index):
"""Retrieves item by the specified index."""
return _GetStructValue(self.values.__getitem__(index))
def __setitem__(self, index, value):
_SetStructValue(self.values.__getitem__(index), value)
def __delitem__(self, key):
del self.values[key]
def _internal_assign(self, elem_seq):
self.Clear()
self.extend(elem_seq)
def _internal_compare(self, other):
size = len(self)
if size != len(other):
return False
for i in range(size):
if isinstance(other[i], (dict, list)):
if not self[i]._internal_compare(other[i]):
return False
elif self[i] != other[i]:
return False
return True
def items(self):
for i in range(len(self)):
yield self[i]
def add_struct(self):
"""Appends and returns a struct value as the next value in the list."""
struct_value = self.values.add().struct_value
# Clear will mark struct_value modified which will indeed create a struct.
struct_value.Clear()
return struct_value
def add_list(self):
"""Appends and returns a list value as the next value in the list."""
list_value = self.values.add().list_value
# Clear will mark list_value modified which will indeed create a list.
list_value.Clear()
return list_value
collections.abc.MutableSequence.register(ListValue)
# LINT.IfChange(wktbases)
WKTBASES = {
'google.protobuf.Any': Any,
'google.protobuf.Duration': Duration,
'google.protobuf.FieldMask': FieldMask,
'google.protobuf.ListValue': ListValue,
'google.protobuf.Struct': Struct,
'google.protobuf.Timestamp': Timestamp,
}
# LINT.ThenChange(//depot/google.protobuf/compiler/python/pyi_generator.cc:wktbases)
@@ -0,0 +1,245 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Constants and static functions to support protocol buffer wire format."""
__author__ = 'robinson@google.com (Will Robinson)'
import struct
from google.protobuf import descriptor
from google.protobuf import message
TAG_TYPE_BITS = 3 # Number of bits used to hold type info in a proto tag.
TAG_TYPE_MASK = (1 << TAG_TYPE_BITS) - 1 # 0x7
# These numbers identify the wire type of a protocol buffer value.
# We use the least-significant TAG_TYPE_BITS bits of the varint-encoded
# tag-and-type to store one of these WIRETYPE_* constants.
# These values must match WireType enum in //google/protobuf/wire_format.h.
WIRETYPE_VARINT = 0
WIRETYPE_FIXED64 = 1
WIRETYPE_LENGTH_DELIMITED = 2
WIRETYPE_START_GROUP = 3
WIRETYPE_END_GROUP = 4
WIRETYPE_FIXED32 = 5
_WIRETYPE_MAX = 5
# Bounds for various integer types.
INT32_MAX = int((1 << 31) - 1)
INT32_MIN = int(-(1 << 31))
UINT32_MAX = (1 << 32) - 1
INT64_MAX = (1 << 63) - 1
INT64_MIN = -(1 << 63)
UINT64_MAX = (1 << 64) - 1
# "struct" format strings that will encode/decode the specified formats.
FORMAT_UINT32_LITTLE_ENDIAN = '<I'
FORMAT_UINT64_LITTLE_ENDIAN = '<Q'
FORMAT_FLOAT_LITTLE_ENDIAN = '<f'
FORMAT_DOUBLE_LITTLE_ENDIAN = '<d'
# We'll have to provide alternate implementations of AppendLittleEndian*() on
# any architectures where these checks fail.
if struct.calcsize(FORMAT_UINT32_LITTLE_ENDIAN) != 4:
raise AssertionError('Format "I" is not a 32-bit number.')
if struct.calcsize(FORMAT_UINT64_LITTLE_ENDIAN) != 8:
raise AssertionError('Format "Q" is not a 64-bit number.')
def PackTag(field_number, wire_type):
"""Returns an unsigned 32-bit integer that encodes the field number and
wire type information in standard protocol message wire format.
Args:
field_number: Expected to be an integer in the range [1, 1 << 29)
wire_type: One of the WIRETYPE_* constants.
"""
if not 0 <= wire_type <= _WIRETYPE_MAX:
raise message.EncodeError('Unknown wire type: %d' % wire_type)
return (field_number << TAG_TYPE_BITS) | wire_type
def UnpackTag(tag):
"""The inverse of PackTag(). Given an unsigned 32-bit number,
returns a (field_number, wire_type) tuple.
"""
return (tag >> TAG_TYPE_BITS), (tag & TAG_TYPE_MASK)
def ZigZagEncode(value):
"""ZigZag Transform: Encodes signed integers so that they can be
effectively used with varint encoding. See wire_format.h for
more details.
"""
if value >= 0:
return value << 1
return (value << 1) ^ (~0)
def ZigZagDecode(value):
"""Inverse of ZigZagEncode()."""
if not value & 0x1:
return value >> 1
return (value >> 1) ^ (~0)
# The *ByteSize() functions below return the number of bytes required to
# serialize "field number + type" information and then serialize the value.
def Int32ByteSize(field_number, int32):
return Int64ByteSize(field_number, int32)
def Int32ByteSizeNoTag(int32):
return _VarUInt64ByteSizeNoTag(0xffffffffffffffff & int32)
def Int64ByteSize(field_number, int64):
# Have to convert to uint before calling UInt64ByteSize().
return UInt64ByteSize(field_number, 0xffffffffffffffff & int64)
def UInt32ByteSize(field_number, uint32):
return UInt64ByteSize(field_number, uint32)
def UInt64ByteSize(field_number, uint64):
return TagByteSize(field_number) + _VarUInt64ByteSizeNoTag(uint64)
def SInt32ByteSize(field_number, int32):
return UInt32ByteSize(field_number, ZigZagEncode(int32))
def SInt64ByteSize(field_number, int64):
return UInt64ByteSize(field_number, ZigZagEncode(int64))
def Fixed32ByteSize(field_number, fixed32):
return TagByteSize(field_number) + 4
def Fixed64ByteSize(field_number, fixed64):
return TagByteSize(field_number) + 8
def SFixed32ByteSize(field_number, sfixed32):
return TagByteSize(field_number) + 4
def SFixed64ByteSize(field_number, sfixed64):
return TagByteSize(field_number) + 8
def FloatByteSize(field_number, flt):
return TagByteSize(field_number) + 4
def DoubleByteSize(field_number, double):
return TagByteSize(field_number) + 8
def BoolByteSize(field_number, b):
return TagByteSize(field_number) + 1
def EnumByteSize(field_number, enum):
return UInt32ByteSize(field_number, enum)
def StringByteSize(field_number, string):
return BytesByteSize(field_number, string.encode('utf-8'))
def BytesByteSize(field_number, b):
return (TagByteSize(field_number)
+ _VarUInt64ByteSizeNoTag(len(b))
+ len(b))
def GroupByteSize(field_number, message):
return (2 * TagByteSize(field_number) # START and END group.
+ message.ByteSize())
def MessageByteSize(field_number, message):
return (TagByteSize(field_number)
+ _VarUInt64ByteSizeNoTag(message.ByteSize())
+ message.ByteSize())
def MessageSetItemByteSize(field_number, msg):
# First compute the sizes of the tags.
# There are 2 tags for the beginning and ending of the repeated group, that
# is field number 1, one with field number 2 (type_id) and one with field
# number 3 (message).
total_size = (2 * TagByteSize(1) + TagByteSize(2) + TagByteSize(3))
# Add the number of bytes for type_id.
total_size += _VarUInt64ByteSizeNoTag(field_number)
message_size = msg.ByteSize()
# The number of bytes for encoding the length of the message.
total_size += _VarUInt64ByteSizeNoTag(message_size)
# The size of the message.
total_size += message_size
return total_size
def TagByteSize(field_number):
"""Returns the bytes required to serialize a tag with this field number."""
# Just pass in type 0, since the type won't affect the tag+type size.
return _VarUInt64ByteSizeNoTag(PackTag(field_number, 0))
# Private helper function for the *ByteSize() functions above.
def _VarUInt64ByteSizeNoTag(uint64):
"""Returns the number of bytes required to serialize a single varint
using boundary value comparisons. (unrolled loop optimization -WPierce)
uint64 must be unsigned.
"""
if uint64 <= 0x7f: return 1
if uint64 <= 0x3fff: return 2
if uint64 <= 0x1fffff: return 3
if uint64 <= 0xfffffff: return 4
if uint64 <= 0x7ffffffff: return 5
if uint64 <= 0x3ffffffffff: return 6
if uint64 <= 0x1ffffffffffff: return 7
if uint64 <= 0xffffffffffffff: return 8
if uint64 <= 0x7fffffffffffffff: return 9
if uint64 > UINT64_MAX:
raise message.EncodeError('Value out of range: %d' % uint64)
return 10
NON_PACKABLE_TYPES = (
descriptor.FieldDescriptor.TYPE_STRING,
descriptor.FieldDescriptor.TYPE_GROUP,
descriptor.FieldDescriptor.TYPE_MESSAGE,
descriptor.FieldDescriptor.TYPE_BYTES
)
def IsTypePackable(field_type):
"""Return true iff packable = true is valid for fields of this type.
Args:
field_type: a FieldDescriptor::Type value.
Returns:
True iff fields of this type are packable.
"""
return field_type not in NON_PACKABLE_TYPES
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,448 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
# TODO: We should just make these methods all "pure-virtual" and move
# all implementation out, into reflection.py for now.
"""Contains an abstract base class for protocol messages."""
__author__ = 'robinson@google.com (Will Robinson)'
_INCONSISTENT_MESSAGE_ATTRIBUTES = ('Extensions',)
class Error(Exception):
"""Base error type for this module."""
pass
class DecodeError(Error):
"""Exception raised when deserializing messages."""
pass
class EncodeError(Error):
"""Exception raised when serializing messages."""
pass
class Message(object):
"""Abstract base class for protocol messages.
Protocol message classes are almost always generated by the protocol
compiler. These generated types subclass Message and implement the methods
shown below.
"""
# TODO: Link to an HTML document here.
# TODO: Document that instances of this class will also
# have an Extensions attribute with __getitem__ and __setitem__.
# Again, not sure how to best convey this.
# TODO: Document these fields and methods.
__slots__ = []
#: The :class:`google.protobuf.Descriptor`
# for this message type.
DESCRIPTOR = None
def __deepcopy__(self, memo=None):
clone = type(self)()
clone.MergeFrom(self)
return clone
def __dir__(self):
"""Provides the list of all accessible Message attributes."""
message_attributes = set(super().__dir__())
# TODO: Remove this once the UPB implementation is improved.
# The UPB proto implementation currently doesn't provide proto fields as
# attributes and they have to added.
if self.DESCRIPTOR is not None:
for field in self.DESCRIPTOR.fields:
message_attributes.add(field.name)
# The Fast C++ proto implementation provides inaccessible attributes that
# have to be removed.
for attribute in _INCONSISTENT_MESSAGE_ATTRIBUTES:
if attribute not in message_attributes:
continue
try:
getattr(self, attribute)
except AttributeError:
message_attributes.remove(attribute)
return sorted(message_attributes)
def __eq__(self, other_msg):
"""Recursively compares two messages by value and structure."""
raise NotImplementedError
def __ne__(self, other_msg):
# Can't just say self != other_msg, since that would infinitely recurse. :)
return not self == other_msg
def __hash__(self):
raise TypeError('unhashable object')
def __str__(self):
"""Outputs a human-readable representation of the message."""
raise NotImplementedError
def __unicode__(self):
"""Outputs a human-readable representation of the message."""
raise NotImplementedError
def __contains__(self, field_name_or_key):
"""Checks if a certain field is set for the message.
Has presence fields return true if the field is set, false if the field is
not set. Fields without presence do raise `ValueError` (this includes
repeated fields, map fields, and implicit presence fields).
If field_name is not defined in the message descriptor, `ValueError` will
be raised.
Note: WKT Struct checks if the key is contained in fields. ListValue checks
if the item is contained in the list.
Args:
field_name_or_key: For Struct, the key (str) of the fields map. For
ListValue, any type that may be contained in the list. For other
messages, name of the field (str) to check for presence.
Returns:
bool: For Struct, whether the item is contained in fields. For ListValue,
whether the item is contained in the list. For other message,
whether a value has been set for the named field.
Raises:
ValueError: For normal messages, if the `field_name_or_key` is not a
member of this message or `field_name_or_key` is not a string.
"""
raise NotImplementedError
def MergeFrom(self, other_msg):
"""Merges the contents of the specified message into current message.
This method merges the contents of the specified message into the current
message. Singular fields that are set in the specified message overwrite
the corresponding fields in the current message. Repeated fields are
appended. Singular sub-messages and groups are recursively merged.
Args:
other_msg (Message): A message to merge into the current message.
"""
raise NotImplementedError
def CopyFrom(self, other_msg):
"""Copies the content of the specified message into the current message.
The method clears the current message and then merges the specified
message using MergeFrom.
Args:
other_msg (Message): A message to copy into the current one.
"""
if self is other_msg:
return
self.Clear()
self.MergeFrom(other_msg)
def Clear(self):
"""Clears all data that was set in the message."""
raise NotImplementedError
def SetInParent(self):
"""Mark this as present in the parent.
This normally happens automatically when you assign a field of a
sub-message, but sometimes you want to make the sub-message
present while keeping it empty. If you find yourself using this,
you may want to reconsider your design.
"""
raise NotImplementedError
def IsInitialized(self):
"""Checks if the message is initialized.
Returns:
bool: The method returns True if the message is initialized (i.e. all of
its required fields are set).
"""
raise NotImplementedError
# TODO: MergeFromString() should probably return None and be
# implemented in terms of a helper that returns the # of bytes read. Our
# deserialization routines would use the helper when recursively
# deserializing, but the end user would almost always just want the no-return
# MergeFromString().
def MergeFromString(self, serialized):
"""Merges serialized protocol buffer data into this message.
When we find a field in `serialized` that is already present
in this message:
- If it's a "repeated" field, we append to the end of our list.
- Else, if it's a scalar, we overwrite our field.
- Else, (it's a nonrepeated composite), we recursively merge
into the existing composite.
Args:
serialized (bytes): Any object that allows us to call
``memoryview(serialized)`` to access a string of bytes using the
buffer interface.
Returns:
int: The number of bytes read from `serialized`.
For non-group messages, this will always be `len(serialized)`,
but for messages which are actually groups, this will
generally be less than `len(serialized)`, since we must
stop when we reach an ``END_GROUP`` tag. Note that if
we *do* stop because of an ``END_GROUP`` tag, the number
of bytes returned does not include the bytes
for the ``END_GROUP`` tag information.
Raises:
DecodeError: if the input cannot be parsed.
"""
# TODO: Document handling of unknown fields.
# TODO: When we switch to a helper, this will return None.
raise NotImplementedError
def ParseFromString(self, serialized):
"""Parse serialized protocol buffer data in binary form into this message.
Like :func:`MergeFromString()`, except we clear the object first.
Raises:
message.DecodeError if the input cannot be parsed.
"""
self.Clear()
return self.MergeFromString(serialized)
def SerializeToString(self, **kwargs):
"""Serializes the protocol message to a binary string.
Keyword Args:
deterministic (bool): If true, requests deterministic serialization
of the protobuf, with predictable ordering of map keys.
Returns:
A binary string representation of the message if all of the required
fields in the message are set (i.e. the message is initialized).
Raises:
EncodeError: if the message isn't initialized (see :func:`IsInitialized`).
"""
raise NotImplementedError
def SerializePartialToString(self, **kwargs):
"""Serializes the protocol message to a binary string.
This method is similar to SerializeToString but doesn't check if the
message is initialized.
Keyword Args:
deterministic (bool): If true, requests deterministic serialization
of the protobuf, with predictable ordering of map keys.
Returns:
bytes: A serialized representation of the partial message.
"""
raise NotImplementedError
# TODO: Decide whether we like these better
# than auto-generated has_foo() and clear_foo() methods
# on the instances themselves. This way is less consistent
# with C++, but it makes reflection-type access easier and
# reduces the number of magically autogenerated things.
#
# TODO: Be sure to document (and test) exactly
# which field names are accepted here. Are we case-sensitive?
# What do we do with fields that share names with Python keywords
# like 'lambda' and 'yield'?
#
# nnorwitz says:
# """
# Typically (in python), an underscore is appended to names that are
# keywords. So they would become lambda_ or yield_.
# """
def ListFields(self):
"""Returns a list of (FieldDescriptor, value) tuples for present fields.
A message field is non-empty if HasField() would return true. A singular
primitive field is non-empty if HasField() would return true in proto2 or it
is non zero in proto3. A repeated field is non-empty if it contains at least
one element. The fields are ordered by field number.
Returns:
list[tuple(FieldDescriptor, value)]: field descriptors and values
for all fields in the message which are not empty. The values vary by
field type.
"""
raise NotImplementedError
def HasField(self, field_name):
"""Checks if a certain field is set for the message.
For a oneof group, checks if any field inside is set. Note that if the
field_name is not defined in the message descriptor, :exc:`ValueError` will
be raised.
Args:
field_name (str): The name of the field to check for presence.
Returns:
bool: Whether a value has been set for the named field.
Raises:
ValueError: if the `field_name` is not a member of this message.
"""
raise NotImplementedError
def ClearField(self, field_name):
"""Clears the contents of a given field.
Inside a oneof group, clears the field set. If the name neither refers to a
defined field or oneof group, :exc:`ValueError` is raised.
Args:
field_name (str): The name of the field to check for presence.
Raises:
ValueError: if the `field_name` is not a member of this message.
"""
raise NotImplementedError
def WhichOneof(self, oneof_group):
"""Returns the name of the field that is set inside a oneof group.
If no field is set, returns None.
Args:
oneof_group (str): the name of the oneof group to check.
Returns:
str or None: The name of the group that is set, or None.
Raises:
ValueError: no group with the given name exists
"""
raise NotImplementedError
def HasExtension(self, field_descriptor):
"""Checks if a certain extension is present for this message.
Extensions are retrieved using the :attr:`Extensions` mapping (if present).
Args:
field_descriptor: The field descriptor for the extension to check.
Returns:
bool: Whether the extension is present for this message.
Raises:
KeyError: if the extension is repeated. Similar to repeated fields,
there is no separate notion of presence: a "not present" repeated
extension is an empty list.
"""
raise NotImplementedError
def ClearExtension(self, field_descriptor):
"""Clears the contents of a given extension.
Args:
field_descriptor: The field descriptor for the extension to clear.
"""
raise NotImplementedError
def UnknownFields(self):
"""Returns the UnknownFieldSet.
Returns:
UnknownFieldSet: The unknown fields stored in this message.
"""
raise NotImplementedError
def DiscardUnknownFields(self):
"""Clears all fields in the :class:`UnknownFieldSet`.
This operation is recursive for nested message.
"""
raise NotImplementedError
def ByteSize(self):
"""Returns the serialized size of this message.
Recursively calls ByteSize() on all contained messages.
Returns:
int: The number of bytes required to serialize this message.
"""
raise NotImplementedError
@classmethod
def FromString(cls, s):
raise NotImplementedError
def _SetListener(self, message_listener):
"""Internal method used by the protocol message implementation.
Clients should not call this directly.
Sets a listener that this message will call on certain state transitions.
The purpose of this method is to register back-edges from children to
parents at runtime, for the purpose of setting "has" bits and
byte-size-dirty bits in the parent and ancestor objects whenever a child or
descendant object is modified.
If the client wants to disconnect this Message from the object tree, she
explicitly sets callback to None.
If message_listener is None, unregisters any existing listener. Otherwise,
message_listener must implement the MessageListener interface in
internal/message_listener.py, and we discard any listener registered
via a previous _SetListener() call.
"""
raise NotImplementedError
def __getstate__(self):
"""Support the pickle protocol."""
return dict(serialized=self.SerializePartialToString())
def __setstate__(self, state):
"""Support the pickle protocol."""
self.__init__()
serialized = state['serialized']
# On Python 3, using encoding='latin1' is required for unpickling
# protos pickled by Python 2.
if not isinstance(serialized, bytes):
serialized = serialized.encode('latin1')
self.ParseFromString(serialized)
def __reduce__(self):
message_descriptor = self.DESCRIPTOR
if message_descriptor.containing_type is None:
return type(self), (), self.__getstate__()
# the message type must be nested.
# Python does not pickle nested classes; use the symbol_database on the
# receiving end.
container = message_descriptor
return (_InternalConstructMessage, (container.full_name,),
self.__getstate__())
def _InternalConstructMessage(full_name):
"""Constructs a nested message."""
from google.protobuf import symbol_database # pylint:disable=g-import-not-at-top
return symbol_database.Default().GetSymbol(full_name)()
@@ -0,0 +1,190 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Provides a factory class for generating dynamic messages.
The easiest way to use this class is if you have access to the FileDescriptor
protos containing the messages you want to create you can just do the following:
message_classes = message_factory.GetMessages(iterable_of_file_descriptors)
my_proto_instance = message_classes['some.proto.package.MessageName']()
"""
__author__ = 'matthewtoia@google.com (Matt Toia)'
import warnings
from google.protobuf import descriptor_pool
from google.protobuf import message
from google.protobuf.internal import api_implementation
if api_implementation.Type() == 'python':
from google.protobuf.internal import python_message as message_impl
else:
from google.protobuf.pyext import cpp_message as message_impl # pylint: disable=g-import-not-at-top
# The type of all Message classes.
_GENERATED_PROTOCOL_MESSAGE_TYPE = message_impl.GeneratedProtocolMessageType
def GetMessageClass(descriptor):
"""Obtains a proto2 message class based on the passed in descriptor.
Passing a descriptor with a fully qualified name matching a previous
invocation will cause the same class to be returned.
Args:
descriptor: The descriptor to build from.
Returns:
A class describing the passed in descriptor.
"""
concrete_class = getattr(descriptor, '_concrete_class', None)
if concrete_class:
return concrete_class
return _InternalCreateMessageClass(descriptor)
def GetMessageClassesForFiles(files, pool):
"""Gets all the messages from specified files.
This will find and resolve dependencies, failing if the descriptor
pool cannot satisfy them.
This will not return the classes for nested types within those classes, for
those, use GetMessageClass() on the nested types within their containing
messages.
For example, for the message:
message NestedTypeMessage {
message NestedType {
string data = 1;
}
NestedType nested = 1;
}
NestedTypeMessage will be in the result, but not
NestedTypeMessage.NestedType.
Args:
files: The file names to extract messages from.
pool: The descriptor pool to find the files including the dependent files.
Returns:
A dictionary mapping proto names to the message classes.
"""
result = {}
for file_name in files:
file_desc = pool.FindFileByName(file_name)
for desc in file_desc.message_types_by_name.values():
result[desc.full_name] = GetMessageClass(desc)
# While the extension FieldDescriptors are created by the descriptor pool,
# the python classes created in the factory need them to be registered
# explicitly, which is done below.
#
# The call to RegisterExtension will specifically check if the
# extension was already registered on the object and either
# ignore the registration if the original was the same, or raise
# an error if they were different.
for extension in file_desc.extensions_by_name.values():
_ = GetMessageClass(extension.containing_type)
if api_implementation.Type() != 'python':
# TODO: Remove this check here. Duplicate extension
# register check should be in descriptor_pool.
if extension is not pool.FindExtensionByNumber(
extension.containing_type, extension.number
):
raise ValueError('Double registration of Extensions')
# Recursively load protos for extension field, in order to be able to
# fully represent the extension. This matches the behavior for regular
# fields too.
if extension.message_type:
GetMessageClass(extension.message_type)
return result
def _InternalCreateMessageClass(descriptor):
"""Builds a proto2 message class based on the passed in descriptor.
Args:
descriptor: The descriptor to build from.
Returns:
A class describing the passed in descriptor.
"""
descriptor_name = descriptor.name
result_class = _GENERATED_PROTOCOL_MESSAGE_TYPE(
descriptor_name,
(message.Message,),
{
'DESCRIPTOR': descriptor,
# If module not set, it wrongly points to message_factory module.
'__module__': None,
},
)
for field in descriptor.fields:
if field.message_type:
GetMessageClass(field.message_type)
for extension in result_class.DESCRIPTOR.extensions:
extended_class = GetMessageClass(extension.containing_type)
if api_implementation.Type() != 'python':
# TODO: Remove this check here. Duplicate extension
# register check should be in descriptor_pool.
pool = extension.containing_type.file.pool
if extension is not pool.FindExtensionByNumber(
extension.containing_type, extension.number
):
raise ValueError('Double registration of Extensions')
if extension.message_type:
GetMessageClass(extension.message_type)
return result_class
# Deprecated. Please use GetMessageClass() or GetMessageClassesForFiles()
# method above instead.
class MessageFactory(object):
"""Factory for creating Proto2 messages from descriptors in a pool."""
def __init__(self, pool=None):
"""Initializes a new factory."""
self.pool = pool or descriptor_pool.DescriptorPool()
def GetMessages(file_protos, pool=None):
"""Builds a dictionary of all the messages available in a set of files.
Args:
file_protos: Iterable of FileDescriptorProto to build messages out of.
pool: The descriptor pool to add the file protos.
Returns:
A dictionary mapping proto names to the message classes. This will include
any dependent messages as well as any messages defined in the same file as
a specified message.
"""
# The cpp implementation of the protocol buffer library requires to add the
# message in topological order of the dependency graph.
des_pool = pool or descriptor_pool.DescriptorPool()
file_by_name = {file_proto.name: file_proto for file_proto in file_protos}
def _AddFile(file_proto):
for dependency in file_proto.dependency:
if dependency in file_by_name:
# Remove from elements to be visited, in order to cut cycles.
_AddFile(file_by_name.pop(dependency))
des_pool.Add(file_proto)
while file_by_name:
_AddFile(file_by_name.popitem()[1])
return GetMessageClassesForFiles(
[file_proto.name for file_proto in file_protos], des_pool
)
@@ -0,0 +1,153 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Contains the Nextgen Pythonic protobuf APIs."""
import io
from typing import Text, Type, TypeVar
from google.protobuf.internal import decoder
from google.protobuf.internal import encoder
from google.protobuf.message import Message
_MESSAGE = TypeVar('_MESSAGE', bound='Message')
def serialize(message: _MESSAGE, deterministic: bool = None) -> bytes:
"""Return the serialized proto.
Args:
message: The proto message to be serialized.
deterministic: If true, requests deterministic serialization
of the protobuf, with predictable ordering of map keys.
Returns:
A binary bytes representation of the message.
"""
return message.SerializeToString(deterministic=deterministic)
def parse(message_class: Type[_MESSAGE], payload: bytes) -> _MESSAGE:
"""Given a serialized data in binary form, deserialize it into a Message.
Args:
message_class: The message meta class.
payload: A serialized bytes in binary form.
Returns:
A new message deserialized from payload.
"""
new_message = message_class()
new_message.ParseFromString(payload)
return new_message
def serialize_length_prefixed(message: _MESSAGE, output: io.BytesIO) -> None:
"""Writes the size of the message as a varint and the serialized message.
Writes the size of the message as a varint and then the serialized message.
This allows more data to be written to the output after the message. Use
parse_length_prefixed to parse messages written by this method.
The output stream must be buffered, e.g. using
https://docs.python.org/3/library/io.html#buffered-streams.
Example usage:
out = io.BytesIO()
for msg in message_list:
proto.serialize_length_prefixed(msg, out)
Args:
message: The protocol buffer message that should be serialized.
output: BytesIO or custom buffered IO that data should be written to.
"""
size = message.ByteSize()
encoder._VarintEncoder()(output.write, size)
out_size = output.write(serialize(message))
if out_size != size:
raise TypeError(
'Failed to write complete message (wrote: %d, expected: %d)'
'. Ensure output is using buffered IO.' % (out_size, size)
)
def parse_length_prefixed(
message_class: Type[_MESSAGE], input_bytes: io.BytesIO
) -> _MESSAGE:
"""Parse a message from input_bytes.
Args:
message_class: The protocol buffer message class that parser should parse.
input_bytes: A buffered input.
Example usage:
while True:
msg = proto.parse_length_prefixed(message_class, input_bytes)
if msg is None:
break
...
Returns:
A parsed message if successful. None if input_bytes is at EOF.
"""
size = decoder._DecodeVarint(input_bytes)
if size is None:
# It is the end of buffered input. See example usage in the
# API description.
return None
message = message_class()
if size == 0:
return message
parsed_size = message.ParseFromString(input_bytes.read(size))
if parsed_size != size:
raise ValueError(
'Truncated message or non-buffered input_bytes: '
'Expected {0} bytes but only {1} bytes parsed for '
'{2}.'.format(size, parsed_size, message.DESCRIPTOR.name)
)
return message
def byte_size(message: Message) -> int:
"""Returns the serialized size of this message.
Args:
message: A proto message.
Returns:
int: The number of bytes required to serialize this message.
"""
return message.ByteSize()
def clear_message(message: Message) -> None:
"""Clears all data that was set in the message.
Args:
message: The proto message to be cleared.
"""
message.Clear()
def clear_field(message: Message, field_name: Text) -> None:
"""Clears the contents of a given field.
Inside a oneof group, clears the field set. If the name neither refers to a
defined field or oneof group, :exc:`ValueError` is raised.
Args:
message: The proto message.
field_name (str): The name of the field to be cleared.
Raises:
ValueError: if the `field_name` is not a member of this message.
"""
message.ClearField(field_name)
@@ -0,0 +1,111 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Dynamic Protobuf class creator."""
from collections import OrderedDict
import hashlib
import os
from google.protobuf import descriptor_pb2
from google.protobuf import descriptor
from google.protobuf import descriptor_pool
from google.protobuf import message_factory
def _GetMessageFromFactory(pool, full_name):
"""Get a proto class from the MessageFactory by name.
Args:
pool: a descriptor pool.
full_name: str, the fully qualified name of the proto type.
Returns:
A class, for the type identified by full_name.
Raises:
KeyError, if the proto is not found in the factory's descriptor pool.
"""
proto_descriptor = pool.FindMessageTypeByName(full_name)
proto_cls = message_factory.GetMessageClass(proto_descriptor)
return proto_cls
def MakeSimpleProtoClass(fields, full_name=None, pool=None):
"""Create a Protobuf class whose fields are basic types.
Note: this doesn't validate field names!
Args:
fields: dict of {name: field_type} mappings for each field in the proto. If
this is an OrderedDict the order will be maintained, otherwise the
fields will be sorted by name.
full_name: optional str, the fully-qualified name of the proto type.
pool: optional DescriptorPool instance.
Returns:
a class, the new protobuf class with a FileDescriptor.
"""
pool_instance = pool or descriptor_pool.DescriptorPool()
if full_name is not None:
try:
proto_cls = _GetMessageFromFactory(pool_instance, full_name)
return proto_cls
except KeyError:
# The factory's DescriptorPool doesn't know about this class yet.
pass
# Get a list of (name, field_type) tuples from the fields dict. If fields was
# an OrderedDict we keep the order, but otherwise we sort the field to ensure
# consistent ordering.
field_items = fields.items()
if not isinstance(fields, OrderedDict):
field_items = sorted(field_items)
# Use a consistent file name that is unlikely to conflict with any imported
# proto files.
fields_hash = hashlib.sha1()
for f_name, f_type in field_items:
fields_hash.update(f_name.encode('utf-8'))
fields_hash.update(str(f_type).encode('utf-8'))
proto_file_name = fields_hash.hexdigest() + '.proto'
# If the proto is anonymous, use the same hash to name it.
if full_name is None:
full_name = ('net.proto2.python.public.proto_builder.AnonymousProto_' +
fields_hash.hexdigest())
try:
proto_cls = _GetMessageFromFactory(pool_instance, full_name)
return proto_cls
except KeyError:
# The factory's DescriptorPool doesn't know about this class yet.
pass
# This is the first time we see this proto: add a new descriptor to the pool.
pool_instance.Add(
_MakeFileDescriptorProto(proto_file_name, full_name, field_items))
return _GetMessageFromFactory(pool_instance, full_name)
def _MakeFileDescriptorProto(proto_file_name, full_name, field_items):
"""Populate FileDescriptorProto for MessageFactory's DescriptorPool."""
package, name = full_name.rsplit('.', 1)
file_proto = descriptor_pb2.FileDescriptorProto()
file_proto.name = os.path.join(package.replace('.', '/'), proto_file_name)
file_proto.package = package
desc_proto = file_proto.message_type.add()
desc_proto.name = name
for f_number, (f_name, f_type) in enumerate(field_items, 1):
field_proto = desc_proto.field.add()
field_proto.name = f_name
# # If the number falls in the reserved range, reassign it to the correct
# # number after the range.
if f_number >= descriptor.FieldDescriptor.FIRST_RESERVED_FIELD_NUMBER:
f_number += (
descriptor.FieldDescriptor.LAST_RESERVED_FIELD_NUMBER -
descriptor.FieldDescriptor.FIRST_RESERVED_FIELD_NUMBER + 1)
field_proto.number = f_number
field_proto.label = descriptor_pb2.FieldDescriptorProto.LABEL_OPTIONAL
field_proto.type = f_type
return file_proto
@@ -0,0 +1,83 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Contains the Nextgen Pythonic Protobuf JSON APIs."""
from typing import Optional, Type
from google.protobuf.message import Message
from google.protobuf.descriptor_pool import DescriptorPool
from google.protobuf import json_format
def serialize(
message: Message,
always_print_fields_with_no_presence: bool=False,
preserving_proto_field_name: bool=False,
use_integers_for_enums: bool=False,
descriptor_pool: Optional[DescriptorPool]=None,
float_precision: int=None,
) -> dict:
"""Converts protobuf message to a dictionary.
When the dictionary is encoded to JSON, it conforms to proto3 JSON spec.
Args:
message: The protocol buffers message instance to serialize.
always_print_fields_with_no_presence: If True, fields without
presence (implicit presence scalars, repeated fields, and map fields) will
always be serialized. Any field that supports presence is not affected by
this option (including singular message fields and oneof fields).
preserving_proto_field_name: If True, use the original proto field names as
defined in the .proto file. If False, convert the field names to
lowerCamelCase.
use_integers_for_enums: If true, print integers instead of enum names.
descriptor_pool: A Descriptor Pool for resolving types. If None use the
default.
float_precision: If set, use this to specify float field valid digits.
Returns:
A dict representation of the protocol buffer message.
"""
return json_format.MessageToDict(
message,
always_print_fields_with_no_presence=always_print_fields_with_no_presence,
preserving_proto_field_name=preserving_proto_field_name,
use_integers_for_enums=use_integers_for_enums,
float_precision=float_precision,
)
def parse(
message_class: Type[Message],
js_dict: dict,
ignore_unknown_fields: bool=False,
descriptor_pool: Optional[DescriptorPool]=None,
max_recursion_depth: int=100
) -> Message:
"""Parses a JSON dictionary representation into a message.
Args:
message_class: The message meta class.
js_dict: Dict representation of a JSON message.
ignore_unknown_fields: If True, do not raise errors for unknown fields.
descriptor_pool: A Descriptor Pool for resolving types. If None use the
default.
max_recursion_depth: max recursion depth of JSON message to be deserialized.
JSON messages over this depth will fail to be deserialized. Default value
is 100.
Returns:
A new message passed from json_dict.
"""
new_message = message_class()
json_format.ParseDict(
js_dict=js_dict,
message=new_message,
ignore_unknown_fields=ignore_unknown_fields,
descriptor_pool=descriptor_pool,
max_recursion_depth=max_recursion_depth,
)
return new_message
@@ -0,0 +1,129 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2025 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Contains the Nextgen Pythonic Protobuf Text Format APIs."""
from typing import AnyStr, Callable, Optional, Text, Type, Union
from google.protobuf import text_format
from google.protobuf.descriptor_pool import DescriptorPool
from google.protobuf.message import Message
_MsgFormatter = Callable[[Message, Union[int, bool], bool], Optional[Text]]
def serialize(
message: Message,
as_utf8: bool = True,
as_one_line: bool = False,
use_short_repeated_primitives: bool = False,
pointy_brackets: bool = False,
use_index_order: bool = False,
float_format: Optional[str] = None,
double_format: Optional[str] = None,
use_field_number: bool = False,
descriptor_pool: Optional[DescriptorPool] = None,
indent: int = 0,
message_formatter: Optional[_MsgFormatter] = None,
print_unknown_fields: bool = False,
force_colon: bool = False,
) -> str:
"""Convert protobuf message to text format.
Double values can be formatted compactly with 15 digits of
precision (which is the most that IEEE 754 "double" can guarantee)
using double_format='.15g'. To ensure that converting to text and back to a
proto will result in an identical value, double_format='.17g' should be used.
Args:
message: The protocol buffers message.
as_utf8: Return unescaped Unicode for non-ASCII characters.
as_one_line: Don't introduce newlines between fields.
use_short_repeated_primitives: Use short repeated format for primitives.
pointy_brackets: If True, use angle brackets instead of curly braces for
nesting.
use_index_order: If True, fields of a proto message will be printed using
the order defined in source code instead of the field number, extensions
will be printed at the end of the message and their relative order is
determined by the extension number. By default, use the field number
order.
float_format (str): If set, use this to specify float field formatting (per
the "Format Specification Mini-Language"); otherwise, shortest float that
has same value in wire will be printed. Also affect double field if
double_format is not set but float_format is set.
double_format (str): If set, use this to specify double field formatting
(per the "Format Specification Mini-Language"); if it is not set but
float_format is set, use float_format. Otherwise, use ``str()``
use_field_number: If True, print field numbers instead of names.
descriptor_pool (DescriptorPool): Descriptor pool used to resolve Any types.
indent (int): The initial indent level, in terms of spaces, for pretty
print.
message_formatter (function(message, indent, as_one_line) -> unicode|None):
Custom formatter for selected sub-messages (usually based on message
type). Use to pretty print parts of the protobuf for easier diffing.
print_unknown_fields: If True, unknown fields will be printed.
force_colon: If set, a colon will be added after the field name even if the
field is a proto message.
Returns:
str: A string of the text formatted protocol buffer message.
"""
return text_format.MessageToString(
message=message,
as_utf8=as_utf8,
as_one_line=as_one_line,
use_short_repeated_primitives=use_short_repeated_primitives,
pointy_brackets=pointy_brackets,
use_index_order=use_index_order,
float_format=float_format,
double_format=double_format,
use_field_number=use_field_number,
descriptor_pool=descriptor_pool,
indent=indent,
message_formatter=message_formatter,
print_unknown_fields=print_unknown_fields,
force_colon=force_colon,
)
def parse(
message_class: Type[Message],
text: AnyStr,
allow_unknown_extension: bool = False,
allow_field_number: bool = False,
descriptor_pool: Optional[DescriptorPool] = None,
allow_unknown_field: bool = False,
) -> Message:
"""Parses a text representation of a protocol message into a message.
Args:
message_class: The message meta class.
text (str): Message text representation.
message (Message): A protocol buffer message to merge into.
allow_unknown_extension: if True, skip over missing extensions and keep
parsing
allow_field_number: if True, both field number and field name are allowed.
descriptor_pool (DescriptorPool): Descriptor pool used to resolve Any types.
allow_unknown_field: if True, skip over unknown field and keep parsing.
Avoid to use this option if possible. It may hide some errors (e.g.
spelling error on field name)
Returns:
Message: A new message passed from text.
Raises:
ParseError: On text parsing problems.
"""
new_message = message_class()
text_format.Parse(
text=text,
message=new_message,
allow_unknown_extension=allow_unknown_extension,
allow_field_number=allow_field_number,
descriptor_pool=descriptor_pool,
allow_unknown_field=allow_unknown_field,
)
return new_message
@@ -0,0 +1,49 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Protocol message implementation hooks for C++ implementation.
Contains helper functions used to create protocol message classes from
Descriptor objects at runtime backed by the protocol buffer C++ API.
"""
__author__ = 'tibell@google.com (Johan Tibell)'
from google.protobuf.internal import api_implementation
# pylint: disable=protected-access
_message = api_implementation._c_module
# TODO: Remove this import after fix api_implementation
if _message is None:
from google.protobuf.pyext import _message
class GeneratedProtocolMessageType(_message.MessageMeta):
"""Metaclass for protocol message classes created at runtime from Descriptors.
The protocol compiler currently uses this metaclass to create protocol
message classes at runtime. Clients can also manually create their own
classes at runtime, as in this example:
mydescriptor = Descriptor(.....)
factory = symbol_database.Default()
factory.pool.AddDescriptor(mydescriptor)
MyProtoClass = message_factory.GetMessageClass(mydescriptor)
myproto_instance = MyProtoClass()
myproto.foo_field = 23
...
The above example will not work for nested types. If you wish to include them,
use reflection.MakeClass() instead of manually instantiating the class in
order to create the appropriate class structure.
"""
# Must be consistent with the protocol-compiler code in
# proto2/compiler/internal/generator.*.
_DESCRIPTOR_KEY = 'DESCRIPTOR'
@@ -0,0 +1,36 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
# This code is meant to work on Python 2.4 and above only.
"""Contains a metaclass and helper functions used to create
protocol message classes from Descriptor objects at runtime.
Recall that a metaclass is the "type" of a class.
(A class is to a metaclass what an instance is to a class.)
In this case, we use the GeneratedProtocolMessageType metaclass
to inject all the useful functionality into the classes
output by the protocol compiler at compile-time.
The upshot of all this is that the real implementation
details for ALL pure-Python protocol buffers are *here in
this file*.
"""
__author__ = 'robinson@google.com (Will Robinson)'
import warnings
from google.protobuf import message_factory
from google.protobuf import symbol_database
# The type of all Message classes.
# Part of the public interface, but normally only used by message factories.
GeneratedProtocolMessageType = message_factory._GENERATED_PROTOCOL_MESSAGE_TYPE
MESSAGE_CLASS_CACHE = {}
@@ -0,0 +1,104 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Protobuf Runtime versions and validators.
It should only be accessed by Protobuf gencodes and tests. DO NOT USE it
elsewhere.
"""
__author__ = 'shaod@google.com (Dennis Shao)'
from enum import Enum
import os
import warnings
class Domain(Enum):
GOOGLE_INTERNAL = 1
PUBLIC = 2
# The versions of this Python Protobuf runtime to be changed automatically by
# the Protobuf release process. Do not edit them manually.
# These OSS versions are not stripped to avoid merging conflicts.
OSS_DOMAIN = Domain.PUBLIC
OSS_MAJOR = 6
OSS_MINOR = 33
OSS_PATCH = 5
OSS_SUFFIX = ''
DOMAIN = OSS_DOMAIN
MAJOR = OSS_MAJOR
MINOR = OSS_MINOR
PATCH = OSS_PATCH
SUFFIX = OSS_SUFFIX
# Avoid flooding of warnings.
_MAX_WARNING_COUNT = 20
_warning_count = 0
class VersionError(Exception):
"""Exception class for version violation."""
def _ReportVersionError(msg):
raise VersionError(msg)
def ValidateProtobufRuntimeVersion(
gen_domain, gen_major, gen_minor, gen_patch, gen_suffix, location
):
"""Function to validate versions.
Args:
gen_domain: The domain where the code was generated from.
gen_major: The major version number of the gencode.
gen_minor: The minor version number of the gencode.
gen_patch: The patch version number of the gencode.
gen_suffix: The version suffix e.g. '-dev', '-rc1' of the gencode.
location: The proto location that causes the version violation.
Raises:
VersionError: if gencode version is invalid or incompatible with the
runtime.
"""
disable_flag = os.getenv('TEMPORARILY_DISABLE_PROTOBUF_VERSION_CHECK')
if disable_flag is not None and disable_flag.lower() == 'true':
return
global _warning_count
version = f'{MAJOR}.{MINOR}.{PATCH}{SUFFIX}'
gen_version = f'{gen_major}.{gen_minor}.{gen_patch}{gen_suffix}'
if gen_major < 0 or gen_minor < 0 or gen_patch < 0:
raise VersionError(f'Invalid gencode version: {gen_version}')
error_prompt = (
'See Protobuf version guarantees at'
' https://protobuf.dev/support/cross-version-runtime-guarantee.'
)
if gen_domain != DOMAIN:
_ReportVersionError(
'Detected mismatched Protobuf Gencode/Runtime domains when loading'
f' {location}: gencode {gen_domain.name} runtime {DOMAIN.name}.'
' Cross-domain usage of Protobuf is not supported.'
)
if (
MAJOR < gen_major
or (MAJOR == gen_major and MINOR < gen_minor)
or (MAJOR == gen_major and MINOR == gen_minor and PATCH < gen_patch)
):
_ReportVersionError(
'Detected incompatible Protobuf Gencode/Runtime versions when loading'
f' {location}: gencode {gen_version} runtime {version}. Runtime version'
f' cannot be older than the linked gencode version. {error_prompt}'
)
@@ -0,0 +1,272 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Contains metaclasses used to create protocol service and service stub
classes from ServiceDescriptor objects at runtime.
The GeneratedServiceType and GeneratedServiceStubType metaclasses are used to
inject all useful functionality into the classes output by the protocol
compiler at compile-time.
"""
__author__ = 'petar@google.com (Petar Petrov)'
class GeneratedServiceType(type):
"""Metaclass for service classes created at runtime from ServiceDescriptors.
Implementations for all methods described in the Service class are added here
by this class. We also create properties to allow getting/setting all fields
in the protocol message.
The protocol compiler currently uses this metaclass to create protocol service
classes at runtime. Clients can also manually create their own classes at
runtime, as in this example::
mydescriptor = ServiceDescriptor(.....)
class MyProtoService(service.Service):
__metaclass__ = GeneratedServiceType
DESCRIPTOR = mydescriptor
myservice_instance = MyProtoService()
# ...
"""
_DESCRIPTOR_KEY = 'DESCRIPTOR'
def __init__(cls, name, bases, dictionary):
"""Creates a message service class.
Args:
name: Name of the class (ignored, but required by the metaclass
protocol).
bases: Base classes of the class being constructed.
dictionary: The class dictionary of the class being constructed.
dictionary[_DESCRIPTOR_KEY] must contain a ServiceDescriptor object
describing this protocol service type.
"""
# Don't do anything if this class doesn't have a descriptor. This happens
# when a service class is subclassed.
if GeneratedServiceType._DESCRIPTOR_KEY not in dictionary:
return
descriptor = dictionary[GeneratedServiceType._DESCRIPTOR_KEY]
service_builder = _ServiceBuilder(descriptor)
service_builder.BuildService(cls)
cls.DESCRIPTOR = descriptor
class GeneratedServiceStubType(GeneratedServiceType):
"""Metaclass for service stubs created at runtime from ServiceDescriptors.
This class has similar responsibilities as GeneratedServiceType, except that
it creates the service stub classes.
"""
_DESCRIPTOR_KEY = 'DESCRIPTOR'
def __init__(cls, name, bases, dictionary):
"""Creates a message service stub class.
Args:
name: Name of the class (ignored, here).
bases: Base classes of the class being constructed.
dictionary: The class dictionary of the class being constructed.
dictionary[_DESCRIPTOR_KEY] must contain a ServiceDescriptor object
describing this protocol service type.
"""
super(GeneratedServiceStubType, cls).__init__(name, bases, dictionary)
# Don't do anything if this class doesn't have a descriptor. This happens
# when a service stub is subclassed.
if GeneratedServiceStubType._DESCRIPTOR_KEY not in dictionary:
return
descriptor = dictionary[GeneratedServiceStubType._DESCRIPTOR_KEY]
service_stub_builder = _ServiceStubBuilder(descriptor)
service_stub_builder.BuildServiceStub(cls)
class _ServiceBuilder(object):
"""This class constructs a protocol service class using a service descriptor.
Given a service descriptor, this class constructs a class that represents
the specified service descriptor. One service builder instance constructs
exactly one service class. That means all instances of that class share the
same builder.
"""
def __init__(self, service_descriptor):
"""Initializes an instance of the service class builder.
Args:
service_descriptor: ServiceDescriptor to use when constructing the
service class.
"""
self.descriptor = service_descriptor
def BuildService(builder, cls):
"""Constructs the service class.
Args:
cls: The class that will be constructed.
"""
# CallMethod needs to operate with an instance of the Service class. This
# internal wrapper function exists only to be able to pass the service
# instance to the method that does the real CallMethod work.
# Making sure to use exact argument names from the abstract interface in
# service.py to match the type signature
def _WrapCallMethod(self, method_descriptor, rpc_controller, request, done):
return builder._CallMethod(self, method_descriptor, rpc_controller,
request, done)
def _WrapGetRequestClass(self, method_descriptor):
return builder._GetRequestClass(method_descriptor)
def _WrapGetResponseClass(self, method_descriptor):
return builder._GetResponseClass(method_descriptor)
builder.cls = cls
cls.CallMethod = _WrapCallMethod
cls.GetDescriptor = staticmethod(lambda: builder.descriptor)
cls.GetDescriptor.__doc__ = 'Returns the service descriptor.'
cls.GetRequestClass = _WrapGetRequestClass
cls.GetResponseClass = _WrapGetResponseClass
for method in builder.descriptor.methods:
setattr(cls, method.name, builder._GenerateNonImplementedMethod(method))
def _CallMethod(self, srvc, method_descriptor,
rpc_controller, request, callback):
"""Calls the method described by a given method descriptor.
Args:
srvc: Instance of the service for which this method is called.
method_descriptor: Descriptor that represent the method to call.
rpc_controller: RPC controller to use for this method's execution.
request: Request protocol message.
callback: A callback to invoke after the method has completed.
"""
if method_descriptor.containing_service != self.descriptor:
raise RuntimeError(
'CallMethod() given method descriptor for wrong service type.')
method = getattr(srvc, method_descriptor.name)
return method(rpc_controller, request, callback)
def _GetRequestClass(self, method_descriptor):
"""Returns the class of the request protocol message.
Args:
method_descriptor: Descriptor of the method for which to return the
request protocol message class.
Returns:
A class that represents the input protocol message of the specified
method.
"""
if method_descriptor.containing_service != self.descriptor:
raise RuntimeError(
'GetRequestClass() given method descriptor for wrong service type.')
return method_descriptor.input_type._concrete_class
def _GetResponseClass(self, method_descriptor):
"""Returns the class of the response protocol message.
Args:
method_descriptor: Descriptor of the method for which to return the
response protocol message class.
Returns:
A class that represents the output protocol message of the specified
method.
"""
if method_descriptor.containing_service != self.descriptor:
raise RuntimeError(
'GetResponseClass() given method descriptor for wrong service type.')
return method_descriptor.output_type._concrete_class
def _GenerateNonImplementedMethod(self, method):
"""Generates and returns a method that can be set for a service methods.
Args:
method: Descriptor of the service method for which a method is to be
generated.
Returns:
A method that can be added to the service class.
"""
return lambda inst, rpc_controller, request, callback: (
self._NonImplementedMethod(method.name, rpc_controller, callback))
def _NonImplementedMethod(self, method_name, rpc_controller, callback):
"""The body of all methods in the generated service class.
Args:
method_name: Name of the method being executed.
rpc_controller: RPC controller used to execute this method.
callback: A callback which will be invoked when the method finishes.
"""
rpc_controller.SetFailed('Method %s not implemented.' % method_name)
callback(None)
class _ServiceStubBuilder(object):
"""Constructs a protocol service stub class using a service descriptor.
Given a service descriptor, this class constructs a suitable stub class.
A stub is just a type-safe wrapper around an RpcChannel which emulates a
local implementation of the service.
One service stub builder instance constructs exactly one class. It means all
instances of that class share the same service stub builder.
"""
def __init__(self, service_descriptor):
"""Initializes an instance of the service stub class builder.
Args:
service_descriptor: ServiceDescriptor to use when constructing the
stub class.
"""
self.descriptor = service_descriptor
def BuildServiceStub(self, cls):
"""Constructs the stub class.
Args:
cls: The class that will be constructed.
"""
def _ServiceStubInit(stub, rpc_channel):
stub.rpc_channel = rpc_channel
self.cls = cls
cls.__init__ = _ServiceStubInit
for method in self.descriptor.methods:
setattr(cls, method.name, self._GenerateStubMethod(method))
def _GenerateStubMethod(self, method):
return (lambda inst, rpc_controller, request, callback=None:
self._StubMethod(inst, method, rpc_controller, request, callback))
def _StubMethod(self, stub, method_descriptor,
rpc_controller, request, callback):
"""The body of all service methods in the generated stub class.
Args:
stub: Stub instance.
method_descriptor: Descriptor of the invoked method.
rpc_controller: Rpc controller to execute the method.
request: Request protocol message.
callback: A callback to execute when the method finishes.
Returns:
Response message (in case of blocking call).
"""
return stub.rpc_channel.CallMethod(
method_descriptor, rpc_controller, request,
method_descriptor.output_type._concrete_class, callback)
@@ -0,0 +1,37 @@
# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# NO CHECKED-IN PROTOBUF GENCODE
# source: google/protobuf/source_context.proto
# Protobuf Python Version: 6.33.5
"""Generated protocol buffer code."""
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import runtime_version as _runtime_version
from google.protobuf import symbol_database as _symbol_database
from google.protobuf.internal import builder as _builder
_runtime_version.ValidateProtobufRuntimeVersion(
_runtime_version.Domain.PUBLIC,
6,
33,
5,
'',
'google/protobuf/source_context.proto'
)
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n$google/protobuf/source_context.proto\x12\x0fgoogle.protobuf\",\n\rSourceContext\x12\x1b\n\tfile_name\x18\x01 \x01(\tR\x08\x66ileNameB\x8a\x01\n\x13\x63om.google.protobufB\x12SourceContextProtoP\x01Z6google.golang.org/protobuf/types/known/sourcecontextpb\xa2\x02\x03GPB\xaa\x02\x1eGoogle.Protobuf.WellKnownTypesb\x06proto3')
_globals = globals()
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'google.protobuf.source_context_pb2', _globals)
if not _descriptor._USE_C_DESCRIPTORS:
_globals['DESCRIPTOR']._loaded_options = None
_globals['DESCRIPTOR']._serialized_options = b'\n\023com.google.protobufB\022SourceContextProtoP\001Z6google.golang.org/protobuf/types/known/sourcecontextpb\242\002\003GPB\252\002\036Google.Protobuf.WellKnownTypes'
_globals['_SOURCECONTEXT']._serialized_start=57
_globals['_SOURCECONTEXT']._serialized_end=101
# @@protoc_insertion_point(module_scope)
@@ -0,0 +1,47 @@
# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# NO CHECKED-IN PROTOBUF GENCODE
# source: google/protobuf/struct.proto
# Protobuf Python Version: 6.33.5
"""Generated protocol buffer code."""
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import runtime_version as _runtime_version
from google.protobuf import symbol_database as _symbol_database
from google.protobuf.internal import builder as _builder
_runtime_version.ValidateProtobufRuntimeVersion(
_runtime_version.Domain.PUBLIC,
6,
33,
5,
'',
'google/protobuf/struct.proto'
)
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1cgoogle/protobuf/struct.proto\x12\x0fgoogle.protobuf\"\x98\x01\n\x06Struct\x12;\n\x06\x66ields\x18\x01 \x03(\x0b\x32#.google.protobuf.Struct.FieldsEntryR\x06\x66ields\x1aQ\n\x0b\x46ieldsEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12,\n\x05value\x18\x02 \x01(\x0b\x32\x16.google.protobuf.ValueR\x05value:\x02\x38\x01\"\xb2\x02\n\x05Value\x12;\n\nnull_value\x18\x01 \x01(\x0e\x32\x1a.google.protobuf.NullValueH\x00R\tnullValue\x12#\n\x0cnumber_value\x18\x02 \x01(\x01H\x00R\x0bnumberValue\x12#\n\x0cstring_value\x18\x03 \x01(\tH\x00R\x0bstringValue\x12\x1f\n\nbool_value\x18\x04 \x01(\x08H\x00R\tboolValue\x12<\n\x0cstruct_value\x18\x05 \x01(\x0b\x32\x17.google.protobuf.StructH\x00R\x0bstructValue\x12;\n\nlist_value\x18\x06 \x01(\x0b\x32\x1a.google.protobuf.ListValueH\x00R\tlistValueB\x06\n\x04kind\";\n\tListValue\x12.\n\x06values\x18\x01 \x03(\x0b\x32\x16.google.protobuf.ValueR\x06values*\x1b\n\tNullValue\x12\x0e\n\nNULL_VALUE\x10\x00\x42\x7f\n\x13\x63om.google.protobufB\x0bStructProtoP\x01Z/google.golang.org/protobuf/types/known/structpb\xf8\x01\x01\xa2\x02\x03GPB\xaa\x02\x1eGoogle.Protobuf.WellKnownTypesb\x06proto3')
_globals = globals()
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'google.protobuf.struct_pb2', _globals)
if not _descriptor._USE_C_DESCRIPTORS:
_globals['DESCRIPTOR']._loaded_options = None
_globals['DESCRIPTOR']._serialized_options = b'\n\023com.google.protobufB\013StructProtoP\001Z/google.golang.org/protobuf/types/known/structpb\370\001\001\242\002\003GPB\252\002\036Google.Protobuf.WellKnownTypes'
_globals['_STRUCT_FIELDSENTRY']._loaded_options = None
_globals['_STRUCT_FIELDSENTRY']._serialized_options = b'8\001'
_globals['_NULLVALUE']._serialized_start=574
_globals['_NULLVALUE']._serialized_end=601
_globals['_STRUCT']._serialized_start=50
_globals['_STRUCT']._serialized_end=202
_globals['_STRUCT_FIELDSENTRY']._serialized_start=121
_globals['_STRUCT_FIELDSENTRY']._serialized_end=202
_globals['_VALUE']._serialized_start=205
_globals['_VALUE']._serialized_end=511
_globals['_LISTVALUE']._serialized_start=513
_globals['_LISTVALUE']._serialized_end=572
# @@protoc_insertion_point(module_scope)
@@ -0,0 +1,179 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""A database of Python protocol buffer generated symbols.
SymbolDatabase is the MessageFactory for messages generated at compile time,
and makes it easy to create new instances of a registered type, given only the
type's protocol buffer symbol name.
Example usage::
db = symbol_database.SymbolDatabase()
# Register symbols of interest, from one or multiple files.
db.RegisterFileDescriptor(my_proto_pb2.DESCRIPTOR)
db.RegisterMessage(my_proto_pb2.MyMessage)
db.RegisterEnumDescriptor(my_proto_pb2.MyEnum.DESCRIPTOR)
# The database can be used as a MessageFactory, to generate types based on
# their name:
types = db.GetMessages(['my_proto.proto'])
my_message_instance = types['MyMessage']()
# The database's underlying descriptor pool can be queried, so it's not
# necessary to know a type's filename to be able to generate it:
filename = db.pool.FindFileContainingSymbol('MyMessage')
my_message_instance = db.GetMessages([filename])['MyMessage']()
# This functionality is also provided directly via a convenience method:
my_message_instance = db.GetSymbol('MyMessage')()
"""
import warnings
from google.protobuf.internal import api_implementation
from google.protobuf import descriptor_pool
from google.protobuf import message_factory
class SymbolDatabase():
"""A database of Python generated symbols."""
# local cache of registered classes.
_classes = {}
def __init__(self, pool=None):
"""Initializes a new SymbolDatabase."""
self.pool = pool or descriptor_pool.DescriptorPool()
def RegisterMessage(self, message):
"""Registers the given message type in the local database.
Calls to GetSymbol() and GetMessages() will return messages registered here.
Args:
message: A :class:`google.protobuf.message.Message` subclass (or
instance); its descriptor will be registered.
Returns:
The provided message.
"""
desc = message.DESCRIPTOR
self._classes[desc] = message
self.RegisterMessageDescriptor(desc)
return message
def RegisterMessageDescriptor(self, message_descriptor):
"""Registers the given message descriptor in the local database.
Args:
message_descriptor (Descriptor): the message descriptor to add.
"""
if api_implementation.Type() == 'python':
# pylint: disable=protected-access
self.pool._AddDescriptor(message_descriptor)
def RegisterEnumDescriptor(self, enum_descriptor):
"""Registers the given enum descriptor in the local database.
Args:
enum_descriptor (EnumDescriptor): The enum descriptor to register.
Returns:
EnumDescriptor: The provided descriptor.
"""
if api_implementation.Type() == 'python':
# pylint: disable=protected-access
self.pool._AddEnumDescriptor(enum_descriptor)
return enum_descriptor
def RegisterServiceDescriptor(self, service_descriptor):
"""Registers the given service descriptor in the local database.
Args:
service_descriptor (ServiceDescriptor): the service descriptor to
register.
"""
if api_implementation.Type() == 'python':
# pylint: disable=protected-access
self.pool._AddServiceDescriptor(service_descriptor)
def RegisterFileDescriptor(self, file_descriptor):
"""Registers the given file descriptor in the local database.
Args:
file_descriptor (FileDescriptor): The file descriptor to register.
"""
if api_implementation.Type() == 'python':
# pylint: disable=protected-access
self.pool._InternalAddFileDescriptor(file_descriptor)
def GetSymbol(self, symbol):
"""Tries to find a symbol in the local database.
Currently, this method only returns message.Message instances, however, if
may be extended in future to support other symbol types.
Args:
symbol (str): a protocol buffer symbol.
Returns:
A Python class corresponding to the symbol.
Raises:
KeyError: if the symbol could not be found.
"""
return self._classes[self.pool.FindMessageTypeByName(symbol)]
def GetMessages(self, files):
# TODO: Fix the differences with MessageFactory.
"""Gets all registered messages from a specified file.
Only messages already created and registered will be returned; (this is the
case for imported _pb2 modules)
But unlike MessageFactory, this version also returns already defined nested
messages, but does not register any message extensions.
Args:
files (list[str]): The file names to extract messages from.
Returns:
A dictionary mapping proto names to the message classes.
Raises:
KeyError: if a file could not be found.
"""
def _GetAllMessages(desc):
"""Walk a message Descriptor and recursively yields all message names."""
yield desc
for msg_desc in desc.nested_types:
for nested_desc in _GetAllMessages(msg_desc):
yield nested_desc
result = {}
for file_name in files:
file_desc = self.pool.FindFileByName(file_name)
for msg_desc in file_desc.message_types_by_name.values():
for desc in _GetAllMessages(msg_desc):
try:
result[desc.full_name] = self._classes[desc]
except KeyError:
# This descriptor has no registered class, skip it.
pass
return result
_DEFAULT = SymbolDatabase(pool=descriptor_pool.Default())
def Default():
"""Returns the default SymbolDatabase."""
return _DEFAULT
@@ -0,0 +1,106 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Encoding related utilities."""
import re
def _AsciiIsPrint(i):
return i >= 32 and i < 127
def _MakeStrEscapes():
ret = {}
for i in range(0, 128):
if not _AsciiIsPrint(i):
ret[i] = r'\%03o' % i
ret[ord('\t')] = r'\t' # optional escape
ret[ord('\n')] = r'\n' # optional escape
ret[ord('\r')] = r'\r' # optional escape
ret[ord('"')] = r'\"' # necessary escape
ret[ord('\'')] = r"\'" # optional escape
ret[ord('\\')] = r'\\' # necessary escape
return ret
# Maps int -> char, performing string escapes.
_str_escapes = _MakeStrEscapes()
# Maps int -> char, performing byte escaping and string escapes
_byte_escapes = {i: chr(i) for i in range(0, 256)}
_byte_escapes.update(_str_escapes)
_byte_escapes.update({i: r'\%03o' % i for i in range(128, 256)})
def _DecodeUtf8EscapeErrors(text_bytes):
ret = ''
while text_bytes:
try:
ret += text_bytes.decode('utf-8').translate(_str_escapes)
text_bytes = ''
except UnicodeDecodeError as e:
ret += text_bytes[:e.start].decode('utf-8').translate(_str_escapes)
ret += _byte_escapes[text_bytes[e.start]]
text_bytes = text_bytes[e.start+1:]
return ret
def CEscape(text, as_utf8) -> str:
"""Escape a bytes string for use in an text protocol buffer.
Args:
text: A byte string to be escaped.
as_utf8: Specifies if result may contain non-ASCII characters.
In Python 3 this allows unescaped non-ASCII Unicode characters.
In Python 2 the return value will be valid UTF-8 rather than only ASCII.
Returns:
Escaped string (str).
"""
# Python's text.encode() 'string_escape' or 'unicode_escape' codecs do not
# satisfy our needs; they encodes unprintable characters using two-digit hex
# escapes whereas our C++ unescaping function allows hex escapes to be any
# length. So, "\0011".encode('string_escape') ends up being "\\x011", which
# will be decoded in C++ as a single-character string with char code 0x11.
text_is_unicode = isinstance(text, str)
if as_utf8:
if text_is_unicode:
return text.translate(_str_escapes)
else:
return _DecodeUtf8EscapeErrors(text)
else:
if text_is_unicode:
text = text.encode('utf-8')
return ''.join([_byte_escapes[c] for c in text])
_CUNESCAPE_HEX = re.compile(r'(\\+)x([0-9a-fA-F])(?![0-9a-fA-F])')
def CUnescape(text: str) -> bytes:
"""Unescape a text string with C-style escape sequences to UTF-8 bytes.
Args:
text: The data to parse in a str.
Returns:
A byte string.
"""
def ReplaceHex(m):
# Only replace the match if the number of leading back slashes is odd. i.e.
# the slash itself is not escaped.
if len(m.group(1)) & 1:
return m.group(1) + 'x0' + m.group(2)
return m.group(0)
# This is required because the 'string_escape' encoding doesn't
# allow single-digit hex escapes (like '\xf').
result = _CUNESCAPE_HEX.sub(ReplaceHex, text)
# Replaces Unicode escape sequences with their character equivalents.
result = result.encode('raw_unicode_escape').decode('raw_unicode_escape')
# Encode Unicode characters as UTF-8, then decode to Latin-1 escaping
# unprintable characters.
result = result.encode('utf-8').decode('unicode_escape')
# Convert Latin-1 text back to a byte string (latin-1 codec also works here).
return result.encode('latin-1')
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,112 @@
# Protocol Buffers - Google's data interchange format
# Copyright 2008 Google Inc. All rights reserved.
#
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file or at
# https://developers.google.com/open-source/licenses/bsd
"""Contains the Timestamp helper APIs."""
import datetime
from typing import Optional
from google.protobuf.timestamp_pb2 import Timestamp
def from_json_string(value: str) -> Timestamp:
"""Parse a RFC 3339 date string format to Timestamp.
Args:
value: A date string. Any fractional digits (or none) and any offset are
accepted as long as they fit into nano-seconds precision. Example of
accepted format: '1972-01-01T10:00:20.021-05:00'
Raises:
ValueError: On parsing problems.
"""
timestamp = Timestamp()
timestamp.FromJsonString(value)
return timestamp
def from_microseconds(micros: float) -> Timestamp:
"""Converts microseconds since epoch to Timestamp."""
timestamp = Timestamp()
timestamp.FromMicroseconds(micros)
return timestamp
def from_milliseconds(millis: float) -> Timestamp:
"""Converts milliseconds since epoch to Timestamp."""
timestamp = Timestamp()
timestamp.FromMilliseconds(millis)
return timestamp
def from_nanoseconds(nanos: float) -> Timestamp:
"""Converts nanoseconds since epoch to Timestamp."""
timestamp = Timestamp()
timestamp.FromNanoseconds(nanos)
return timestamp
def from_seconds(seconds: float) -> Timestamp:
"""Converts seconds since epoch to Timestamp."""
timestamp = Timestamp()
timestamp.FromSeconds(seconds)
return timestamp
def from_current_time() -> Timestamp:
"""Converts the current UTC to Timestamp."""
timestamp = Timestamp()
timestamp.FromDatetime(datetime.datetime.now(tz=datetime.timezone.utc))
return timestamp
def to_json_string(ts: Timestamp) -> str:
"""Converts Timestamp to RFC 3339 date string format.
Returns:
A string converted from timestamp. The string is always Z-normalized
and uses 3, 6 or 9 fractional digits as required to represent the
exact time. Example of the return format: '1972-01-01T10:00:20.021Z'
"""
return ts.ToJsonString()
def to_microseconds(ts: Timestamp) -> int:
"""Converts Timestamp to microseconds since epoch."""
return ts.ToMicroseconds()
def to_milliseconds(ts: Timestamp) -> int:
"""Converts Timestamp to milliseconds since epoch."""
return ts.ToMilliseconds()
def to_nanoseconds(ts: Timestamp) -> int:
"""Converts Timestamp to nanoseconds since epoch."""
return ts.ToNanoseconds()
def to_seconds(ts: Timestamp) -> int:
"""Converts Timestamp to seconds since epoch."""
return ts.ToSeconds()
def to_datetime(
ts: Timestamp, tz: Optional[datetime.tzinfo] = None
) -> datetime.datetime:
"""Converts Timestamp to a datetime.
Args:
tz: A datetime.tzinfo subclass; defaults to None.
Returns:
If tzinfo is None, returns a timezone-naive UTC datetime (with no timezone
information, i.e. not aware that it's UTC).
Otherwise, returns a timezone-aware datetime in the input timezone.
"""
return ts.ToDatetime(tzinfo=tz)
@@ -0,0 +1,37 @@
# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# NO CHECKED-IN PROTOBUF GENCODE
# source: google/protobuf/timestamp.proto
# Protobuf Python Version: 6.33.5
"""Generated protocol buffer code."""
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import runtime_version as _runtime_version
from google.protobuf import symbol_database as _symbol_database
from google.protobuf.internal import builder as _builder
_runtime_version.ValidateProtobufRuntimeVersion(
_runtime_version.Domain.PUBLIC,
6,
33,
5,
'',
'google/protobuf/timestamp.proto'
)
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1fgoogle/protobuf/timestamp.proto\x12\x0fgoogle.protobuf\";\n\tTimestamp\x12\x18\n\x07seconds\x18\x01 \x01(\x03R\x07seconds\x12\x14\n\x05nanos\x18\x02 \x01(\x05R\x05nanosB\x85\x01\n\x13\x63om.google.protobufB\x0eTimestampProtoP\x01Z2google.golang.org/protobuf/types/known/timestamppb\xf8\x01\x01\xa2\x02\x03GPB\xaa\x02\x1eGoogle.Protobuf.WellKnownTypesb\x06proto3')
_globals = globals()
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'google.protobuf.timestamp_pb2', _globals)
if not _descriptor._USE_C_DESCRIPTORS:
_globals['DESCRIPTOR']._loaded_options = None
_globals['DESCRIPTOR']._serialized_options = b'\n\023com.google.protobufB\016TimestampProtoP\001Z2google.golang.org/protobuf/types/known/timestamppb\370\001\001\242\002\003GPB\252\002\036Google.Protobuf.WellKnownTypes'
_globals['_TIMESTAMP']._serialized_start=52
_globals['_TIMESTAMP']._serialized_end=111
# @@protoc_insertion_point(module_scope)
@@ -0,0 +1,53 @@
# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# NO CHECKED-IN PROTOBUF GENCODE
# source: google/protobuf/type.proto
# Protobuf Python Version: 6.33.5
"""Generated protocol buffer code."""
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import runtime_version as _runtime_version
from google.protobuf import symbol_database as _symbol_database
from google.protobuf.internal import builder as _builder
_runtime_version.ValidateProtobufRuntimeVersion(
_runtime_version.Domain.PUBLIC,
6,
33,
5,
'',
'google/protobuf/type.proto'
)
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
from google.protobuf import any_pb2 as google_dot_protobuf_dot_any__pb2
from google.protobuf import source_context_pb2 as google_dot_protobuf_dot_source__context__pb2
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x1agoogle/protobuf/type.proto\x12\x0fgoogle.protobuf\x1a\x19google/protobuf/any.proto\x1a$google/protobuf/source_context.proto\"\xa7\x02\n\x04Type\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\x12.\n\x06\x66ields\x18\x02 \x03(\x0b\x32\x16.google.protobuf.FieldR\x06\x66ields\x12\x16\n\x06oneofs\x18\x03 \x03(\tR\x06oneofs\x12\x31\n\x07options\x18\x04 \x03(\x0b\x32\x17.google.protobuf.OptionR\x07options\x12\x45\n\x0esource_context\x18\x05 \x01(\x0b\x32\x1e.google.protobuf.SourceContextR\rsourceContext\x12/\n\x06syntax\x18\x06 \x01(\x0e\x32\x17.google.protobuf.SyntaxR\x06syntax\x12\x18\n\x07\x65\x64ition\x18\x07 \x01(\tR\x07\x65\x64ition\"\xb4\x06\n\x05\x46ield\x12/\n\x04kind\x18\x01 \x01(\x0e\x32\x1b.google.protobuf.Field.KindR\x04kind\x12\x44\n\x0b\x63\x61rdinality\x18\x02 \x01(\x0e\x32\".google.protobuf.Field.CardinalityR\x0b\x63\x61rdinality\x12\x16\n\x06number\x18\x03 \x01(\x05R\x06number\x12\x12\n\x04name\x18\x04 \x01(\tR\x04name\x12\x19\n\x08type_url\x18\x06 \x01(\tR\x07typeUrl\x12\x1f\n\x0boneof_index\x18\x07 \x01(\x05R\noneofIndex\x12\x16\n\x06packed\x18\x08 \x01(\x08R\x06packed\x12\x31\n\x07options\x18\t \x03(\x0b\x32\x17.google.protobuf.OptionR\x07options\x12\x1b\n\tjson_name\x18\n \x01(\tR\x08jsonName\x12#\n\rdefault_value\x18\x0b \x01(\tR\x0c\x64\x65\x66\x61ultValue\"\xc8\x02\n\x04Kind\x12\x10\n\x0cTYPE_UNKNOWN\x10\x00\x12\x0f\n\x0bTYPE_DOUBLE\x10\x01\x12\x0e\n\nTYPE_FLOAT\x10\x02\x12\x0e\n\nTYPE_INT64\x10\x03\x12\x0f\n\x0bTYPE_UINT64\x10\x04\x12\x0e\n\nTYPE_INT32\x10\x05\x12\x10\n\x0cTYPE_FIXED64\x10\x06\x12\x10\n\x0cTYPE_FIXED32\x10\x07\x12\r\n\tTYPE_BOOL\x10\x08\x12\x0f\n\x0bTYPE_STRING\x10\t\x12\x0e\n\nTYPE_GROUP\x10\n\x12\x10\n\x0cTYPE_MESSAGE\x10\x0b\x12\x0e\n\nTYPE_BYTES\x10\x0c\x12\x0f\n\x0bTYPE_UINT32\x10\r\x12\r\n\tTYPE_ENUM\x10\x0e\x12\x11\n\rTYPE_SFIXED32\x10\x0f\x12\x11\n\rTYPE_SFIXED64\x10\x10\x12\x0f\n\x0bTYPE_SINT32\x10\x11\x12\x0f\n\x0bTYPE_SINT64\x10\x12\"t\n\x0b\x43\x61rdinality\x12\x17\n\x13\x43\x41RDINALITY_UNKNOWN\x10\x00\x12\x18\n\x14\x43\x41RDINALITY_OPTIONAL\x10\x01\x12\x18\n\x14\x43\x41RDINALITY_REQUIRED\x10\x02\x12\x18\n\x14\x43\x41RDINALITY_REPEATED\x10\x03\"\x99\x02\n\x04\x45num\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\x12\x38\n\tenumvalue\x18\x02 \x03(\x0b\x32\x1a.google.protobuf.EnumValueR\tenumvalue\x12\x31\n\x07options\x18\x03 \x03(\x0b\x32\x17.google.protobuf.OptionR\x07options\x12\x45\n\x0esource_context\x18\x04 \x01(\x0b\x32\x1e.google.protobuf.SourceContextR\rsourceContext\x12/\n\x06syntax\x18\x05 \x01(\x0e\x32\x17.google.protobuf.SyntaxR\x06syntax\x12\x18\n\x07\x65\x64ition\x18\x06 \x01(\tR\x07\x65\x64ition\"j\n\tEnumValue\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n\x06number\x18\x02 \x01(\x05R\x06number\x12\x31\n\x07options\x18\x03 \x03(\x0b\x32\x17.google.protobuf.OptionR\x07options\"H\n\x06Option\x12\x12\n\x04name\x18\x01 \x01(\tR\x04name\x12*\n\x05value\x18\x02 \x01(\x0b\x32\x14.google.protobuf.AnyR\x05value*C\n\x06Syntax\x12\x11\n\rSYNTAX_PROTO2\x10\x00\x12\x11\n\rSYNTAX_PROTO3\x10\x01\x12\x13\n\x0fSYNTAX_EDITIONS\x10\x02\x42{\n\x13\x63om.google.protobufB\tTypeProtoP\x01Z-google.golang.org/protobuf/types/known/typepb\xf8\x01\x01\xa2\x02\x03GPB\xaa\x02\x1eGoogle.Protobuf.WellKnownTypesb\x06proto3')
_globals = globals()
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'google.protobuf.type_pb2', _globals)
if not _descriptor._USE_C_DESCRIPTORS:
_globals['DESCRIPTOR']._loaded_options = None
_globals['DESCRIPTOR']._serialized_options = b'\n\023com.google.protobufB\tTypeProtoP\001Z-google.golang.org/protobuf/types/known/typepb\370\001\001\242\002\003GPB\252\002\036Google.Protobuf.WellKnownTypes'
_globals['_SYNTAX']._serialized_start=1699
_globals['_SYNTAX']._serialized_end=1766
_globals['_TYPE']._serialized_start=113
_globals['_TYPE']._serialized_end=408
_globals['_FIELD']._serialized_start=411
_globals['_FIELD']._serialized_end=1231
_globals['_FIELD_KIND']._serialized_start=785
_globals['_FIELD_KIND']._serialized_end=1113
_globals['_FIELD_CARDINALITY']._serialized_start=1115
_globals['_FIELD_CARDINALITY']._serialized_end=1231
_globals['_ENUM']._serialized_start=1234
_globals['_ENUM']._serialized_end=1515
_globals['_ENUMVALUE']._serialized_start=1517
_globals['_ENUMVALUE']._serialized_end=1623
_globals['_OPTION']._serialized_start=1625
_globals['_OPTION']._serialized_end=1697
# @@protoc_insertion_point(module_scope)

Some files were not shown because too many files have changed in this diff Show More