Friday, October 26, 2012

Re: Translation: DjangoUnicodeDecodeError

Hi Guys...

The problem is the gettext is creating only a file with msgid's and msgstr's, but no po header....

for example, my po file is like:
#: .\templates\base.html.py:9
msgid "Title"
msgstr "Main"

#: .\templates\base.html.py:12
msgid "Subtitle"
msgstr "My Cool Page!"

#: .\templates\base.html.py:14
msgid "Presentation"
msgstr "Welcome to Cool Page"

When I add the header:
"Content-Type: text/plain; charset=UTF-8\n" to po file, the problem stopped... The final file is like:

# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER
# This file is distributed under the same license as the PACKAGE package.
# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
#
msgid ""
msgstr ""
"Project-Id-Version: lol\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2009-10-21 12:21+0200\n"
"PO-Revision-Date: 2012-05-27 14:59\n"
"Last-Translator:  <admin@admin.com>\n"
"Language-Team: wow<wow@wow.com\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=3; plural=(n==1 ? 0 : n%10>=2 && n%10<=4 && (n%100<10 || n%100>=20) ? 1 : 2)\n"
"X-Translated-Using:PyCharm 2.6.2\n"

#: .\templates\base.html.py:9
msgid "Title"
msgstr "Main"

#: .\templates\base.html.py:12
msgid "Subtitle"
msgstr "My Cool Page!"

#: .\templates\base.html.py:14
msgid "Presentation"
msgstr "Welcome to Cool Page"

I don't know why this header was not added by default... But probably is a issue on makemessages or in gettext for Windows.

Thanks in advance,

2012/10/25 Sandro Dutra <hexodin@gmail.com>
The makemessages only support creation of *.po file in UTF-8 without BOM. I'm translating the msgstr's in *.po files with PyCharm Editor, not ordinary, PyCharm by default sets the encoding to UTF-8. After I compile the *.po file to *.mo files using compilemessages without no error.

The problem is, when I use non-acsii characters (I called "special chars"), I get the error described bellow, otherwise, if all characters are only ascii default, the translation runs well.

Full traceback:

DjangoUnicodeDecodeError at /

'ascii' codec can't decode byte 0xc3 in position 14: ordinal not in range(128). You passed in <django.utils.functional.__proxy__ object at 0x02892ED0> (<class 'django.utils.functional.__proxy__'>)
Request Method: GET
Request URL: http://127.0.0.1:8000/
Django Version: 1.4.2
Exception Type: DjangoUnicodeDecodeError
Exception Value:
'ascii' codec can't decode byte 0xc3 in position 14: ordinal not in range(128). You passed in <django.utils.functional.__proxy__ object at 0x02892ED0> (<class 'django.utils.functional.__proxy__'>)
Exception Location: C:\Python27\lib\site-packages\django\utils\encoding.py in force_unicode, line 93
Python Executable: C:\Python27\python.exe
Python Version: 2.7.2
Python Path:
['C:\\Users\\sksdutra\\Desktop\\Projetos\\soulmasters',   'C:\\Python27\\lib\\site-packages\\amqplib-1.0.2-py2.7.egg',   'C:\\Python27\\lib\\site-packages\\markdown-2.1.1-py2.7.egg',   'C:\\Python27\\lib\\site-packages\\zope.interface-4.0.1-py2.7-win32.egg',   'C:\\Python27\\lib\\site-packages\\pyopengl-3.0.2b2-py2.7.egg',   'C:\\Python27\\lib\\site-packages\\pyopengl_accelerate-3.0.2b2-py2.7-win32.egg',   'C:\\Python27\\lib\\site-packages\\pip-1.2.1-py2.7.egg',   'C:\\Windows\\system32\\python27.zip',   'C:\\Python27\\DLLs',   'C:\\Python27\\lib',   'C:\\Python27\\lib\\plat-win',   'C:\\Python27\\lib\\lib-tk',   'C:\\Python27',   'C:\\Python27\\lib\\site-packages',   'C:\\Python27\\lib\\site-packages\\PIL',   'C:\\Python27\\lib\\site-packages\\setuptools-0.6c11-py2.7.egg-info']
Server time: Qui, 25 Out 2012 11:03:51 -0300

Unicode error hint

The string that could not be encoded/decoded was: o voc�� dev

Error during template rendering

In template C:\Users\sksdutra\Desktop\Projetos\soulmasters\templates\base.html, error at line 16

ascii

6 <!DOCTYPE html>
7 <html>
8 <head>
9 <title>Soul Masters: {% block title %}{% trans "Title" %}{% endblock %}</title>
10 <meta charset="utf-8" />
11 </head>
12 <body>
13 <h2>Soul Masters: {% trans "Subtitle" %}</h2>
14 <hr />
15 <h2>{% trans "Presentation" %}</h2>
16 <p>{% trans "Presentation Text" %}</p>
17 <hr />
18 <p>Idioma: {{ clang }}<br />
19 {% if isbidi %}
20 Bidirecional: Sim<br />
21 {% else %}
22 Bidirecional: Nao<br />
23 {% endif %}
24 Disponiveis:<br />
25 {% for lang in alangs %}
26 * {{ lang.1 }}<br />

Traceback Switch to copy-and-paste view

  • C:\Python27\lib\site-packages\django\core\handlers\base.py in get_response
    1.                     response = response.render()
      ...
  • C:\Python27\lib\site-packages\django\template\response.py in render
    1.             self._set_content(self.rendered_content)
      ...
  • C:\Python27\lib\site-packages\django\template\response.py in rendered_content
    1.         content = template.render(context)
      ...
  • C:\Python27\lib\site-packages\django\template\base.py in render
    1.             return self._render(context)
      ...
  • C:\Python27\lib\site-packages\django\template\base.py in _render
    1.         return self.nodelist.render(context)
      ...
  • C:\Python27\lib\site-packages\django\template\base.py in render
    1.                 bit = self.render_node(node, context)
      ...
  • C:\Python27\lib\site-packages\django\template\debug.py in render_node
    1.             return node.render(context)
      ...
  • C:\Python27\lib\site-packages\django\templatetags\i18n.py in render
    1.         value = _render_value_in_context(output, context)
      ...
  • C:\Python27\lib\site-packages\django\template\base.py in _render_value_in_context
    1.     value = force_unicode(value)
      ...
  • C:\Python27\lib\site-packages\django\utils\encoding.py in force_unicode
    1.             raise DjangoUnicodeDecodeError(s, *e.args)
      ...

Request information

GET

No GET data

POST

No POST data

FILES

No FILES data

COOKIES

Variable Value
csrftoken
'66EQ3yTOxn4uraX3IaJPCtUSAKUeM7sx'
sessionid
'216c2fd95cadaef01c86ec415728ca74'

META

Variable Value
TMP
'C:\\Users\\sksdutra\\AppData\\Local\\Temp'
PYTHONIOENCODING
'UTF-8'
COMPUTERNAME
'TJ_GPAJUR20'
wsgi.multiprocess
False
RUN_MAIN
'true'
HTTP_COOKIE
'csrftoken=66EQ3yTOxn4uraX3IaJPCtUSAKUeM7sx; sessionid=216c2fd95cadaef01c86ec415728ca74'
USERDOMAIN
'TJMA'
wsgi.multithread
True
AV_APPDATA
'C:\\Users\\sksdutra\\AppData\\Roaming'
SERVER_SOFTWARE
'WSGIServer/0.1 Python/2.7.2'
PSMODULEPATH
'C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\Modules\\'
SCRIPT_NAME
u''
COMMONPROGRAMFILES
'C:\\Program Files\\Common Files'
QUERY_STRING
''
PROCESSOR_IDENTIFIER
'x86 Family 6 Model 30 Stepping 5, GenuineIntel'
REQUEST_METHOD
'GET'
PROGRAMFILES
'C:\\Program Files'
PROCESSOR_REVISION
'1e05'
PATH
'C:\\Program Files\\JetBrains\\PyCharm 2.6.2\\bin\\..\\.\\bin;C:\\Program Files\\NVIDIA Corporation\\PhysX\\Common;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\Program Files\\Common Files\\Lenovo;C:\\Program Files\\Common Files\\Roxio Shared\\10.0\\DLLShared\\;c:\\Python27;c:\\Python27\\Scripts;c:\\php;c:\\php\\ext;C:\\Program Files\\Microsoft SQL Server\\100\\Tools\\Binn\\;C:\\Program Files\\Microsoft SQL Server\\100\\DTS\\Binn\\;C:\\mingw32\\bin;C:\\Program Files\\QuickTime\\QTSystem\\;C:\\Program Files\\TortoiseHg\\;C:\\gettext-utils\\bin\\;C:\\Lua\\5.1;C:\\Lua\\5.1\\clibs;C:\\Program Files\\Common Files\\Nero\\Lib\\;c:\\program files\\jetbrains\\pycharm 2.6.2\\jre\\jre\\bin'
QTJAVA
'C:\\Program Files\\Java\\jre7\\lib\\ext\\QTJava.zip'
SYSTEMROOT
'C:\\Windows'
PYTHONUNBUFFERED
'1'
wsgi.errors
<open file '<stderr>', mode 'w' at 0x019450D0>
CONTENT_LENGTH
''
HTTP_USER_AGENT
'Mozilla/5.0 (Windows NT 6.1; rv:16.0) Gecko/20100101 Firefox/16.0'
DJANGO_SETTINGS_MODULE
'soulmasters.settings'
REMOTE_HOST
''
TEMP
'C:\\Users\\sksdutra\\AppData\\Local\\Temp'
REMOTE_ADDR
'127.0.0.1'
SERVER_PROTOCOL
'HTTP/1.1'
PROCESSOR_ARCHITECTURE
'x86'
wsgi.url_scheme
'http'
ALLUSERSPROFILE
'C:\\ProgramData'
SERVER_PORT
'8000'
LOCALAPPDATA
'C:\\Users\\sksdutra\\AppData\\Local'
HOMEPATH
'\\Users\\sksdutra'
XNAGSV4
'C:\\Program Files\\Microsoft XNA\\XNA Game Studio\\v4.0\\'
HTTP_CONNECTION
'keep-alive'
LUA_PATH
';;C:\\Lua\\5.1\\lua\\?.luac'
XNAGSSHARED
'C:\\Program Files\\Common Files\\Microsoft Shared\\XNA\\'
USERNAME
'sksdutra'
HTTP_ACCEPT
'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
LOGONSERVER
'\\\\SLZWADP14'
COMSPEC
'C:\\Windows\\system32\\cmd.exe'
PROGRAMDATA
'C:\\ProgramData'
PATH_INFO
u'/'
CLASSPATH
'.;C:\\Program Files\\Java\\jre7\\lib\\ext\\QTJava.zip'
CSRF_COOKIE
'66EQ3yTOxn4uraX3IaJPCtUSAKUeM7sx'
wsgi.input
<socket._fileobject object at 0x0281E4F0>
USERDNSDOMAIN
'TJMA.JUS.BR'
HTTP_HOST
'127.0.0.1:8000'
SESSIONNAME
'Console'
PATHEXT
'.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.wlua;.lexe'
CONFIGSETROOT
'C:\\Windows\\ConfigSetRoot'
SWSHARE
'C:\\SWSHARE'
FP_NO_HOST_CHECK
'NO'
WINDIR
'C:\\Windows'
HTTP_ACCEPT_ENCODING
'gzip, deflate'
wsgi.version
(1, 0)
HOMEDRIVE
'C:'
APPDATA
'C:\\Users\\sksdutra\\AppData\\Roaming'
SERVER_NAME
'TJ_GPAJUR20.tjma.jus.br'
wsgi.run_once
False
TVTCOMMON
'C:\\Program Files\\Common Files\\Lenovo'
SYSTEMDRIVE
'C:'
GATEWAY_INTERFACE
'CGI/1.1'
HTTP_ACCEPT_LANGUAGE
'pt-BR,pt;q=0.8,en-US;q=0.5,en;q=0.3'
PYCHARM_HOSTED
'1'
NUMBER_OF_PROCESSORS
'8'
VBOX_INSTALL_PATH
'C:\\Program Files\\Oracle\\VirtualBox\\'
PROCESSOR_LEVEL
'6'
CONTENT_TYPE
'text/plain'
LUA_DEV
'C:\\Lua\\5.1'
wsgi.file_wrapper
''
OS
'Windows_NT'
PUBLIC
'C:\\Users\\Public'
USERPROFILE
'C:\\Users\\sksdutra'

Settings

Using settings module soulmasters.settings

Setting Value
USE_L10N
True
USE_THOUSAND_SEPARATOR
False
CSRF_COOKIE_SECURE
False
LANGUAGE_CODE
'pt-br'
ROOT_URLCONF
'soulmasters.urls'
MANAGERS
(('Sandro Dutra', 'hexodin@gmail.com'),   ('Israel Crisanto', 'israel.crisanto@gmail.com'))
DEFAULT_CHARSET
'utf-8'
STATIC_ROOT
''
MESSAGE_STORAGE
'django.contrib.messages.storage.fallback.FallbackStorage'
EMAIL_SUBJECT_PREFIX
'[Django] '
FILE_UPLOAD_PERMISSIONS
None
URL_VALIDATOR_USER_AGENT
'Django/1.4.2 (https://www.djangoproject.com)'
STATICFILES_FINDERS
('django.contrib.staticfiles.finders.FileSystemFinder',   'django.contrib.staticfiles.finders.AppDirectoriesFinder')
SESSION_COOKIE_DOMAIN
None
SESSION_COOKIE_NAME
'sessionid'
ADMIN_FOR
()
TIME_INPUT_FORMATS
('%H:%M:%S', '%H:%M')
DATABASES
{'default': {'ENGINE': 'django.db.backends.sqlite3',               'HOST': '',               'NAME': 'soulmasters.db',               'OPTIONS': {},               'PASSWORD': u'********************',               'PORT': '',               'TEST_CHARSET': None,               'TEST_COLLATION': None,               'TEST_MIRROR': None,               'TEST_NAME': None,               'TIME_ZONE': 'UTC',               'USER': ''}}
SERVER_EMAIL
'root@localhost'
FILE_UPLOAD_HANDLERS
('django.core.files.uploadhandler.MemoryFileUploadHandler',   'django.core.files.uploadhandler.TemporaryFileUploadHandler')
DEFAULT_CONTENT_TYPE
'text/html'
APPEND_SLASH
True
FIRST_DAY_OF_WEEK
0
DATABASE_ROUTERS
[]
YEAR_MONTH_FORMAT
'F Y'
STATICFILES_STORAGE
'django.contrib.staticfiles.storage.StaticFilesStorage'
CACHES
{'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',               'LOCATION': ''}}
SESSION_COOKIE_PATH
'/'
USE_X_FORWARDED_HOST
False
MIDDLEWARE_CLASSES
('django.middleware.common.CommonMiddleware',   'django.contrib.sessions.middleware.SessionMiddleware',   'django.middleware.csrf.CsrfViewMiddleware',   'django.contrib.auth.middleware.AuthenticationMiddleware',   'django.contrib.messages.middleware.MessageMiddleware')
USE_I18N
True
THOUSAND_SEPARATOR
','
SECRET_KEY
u'********************'
LANGUAGE_COOKIE_NAME
'django_language'
DEFAULT_INDEX_TABLESPACE
''
TRANSACTIONS_MANAGED
False
LOGGING_CONFIG
'django.utils.log.dictConfig'
SEND_BROKEN_LINK_EMAILS
False
TEMPLATE_LOADERS
('django.template.loaders.filesystem.Loader',   'django.template.loaders.app_directories.Loader')
WSGI_APPLICATION
'soulmasters.wsgi.application'
TEMPLATE_DEBUG
True
X_FRAME_OPTIONS
'SAMEORIGIN'
AUTHENTICATION_BACKENDS
('django.contrib.auth.backends.ModelBackend',)
FORCE_SCRIPT_NAME
None
CACHE_BACKEND
'locmem://'
SIGNING_BACKEND
'django.core.signing.TimestampSigner'
SESSION_COOKIE_SECURE
False
CSRF_COOKIE_DOMAIN
None
FILE_CHARSET
'utf-8'
DEBUG
True
SESSION_FILE_PATH
None
DEFAULT_FILE_STORAGE
'django.core.files.storage.FileSystemStorage'
INSTALLED_APPS
('django.contrib.auth',   'django.contrib.contenttypes',   'django.contrib.sessions',   'django.contrib.sites',   'django.contrib.messages',   'django.contrib.staticfiles',   'django.contrib.admin',   'smgame')
LANGUAGES
(('pt-br', 'Portuguese'), ('en', 'English'))
COMMENTS_ALLOW_PROFANITIES
False
STATICFILES_DIRS
()
PREPEND_WWW
False
SECURE_PROXY_SSL_HEADER
None
SESSION_COOKIE_HTTPONLY
True
DEBUG_PROPAGATE_EXCEPTIONS
False
MONTH_DAY_FORMAT
'F j'
LOGIN_URL
'/accounts/login/'
SESSION_EXPIRE_AT_BROWSER_CLOSE
False
TIME_FORMAT
'P'
DATE_INPUT_FORMATS
('%Y-%m-%d',   '%m/%d/%Y',   '%m/%d/%y',   '%b %d %Y',   '%b %d, %Y',   '%d %b %Y',   '%d %b, %Y',   '%B %d %Y',   '%B %d, %Y',   '%d %B %Y',   '%d %B, %Y')
CSRF_COOKIE_NAME
'csrftoken'
EMAIL_HOST_PASSWORD
u'********************'
PASSWORD_RESET_TIMEOUT_DAYS
u'********************'
CACHE_MIDDLEWARE_ALIAS
'default'
SESSION_SAVE_EVERY_REQUEST
False
NUMBER_GROUPING
0
SESSION_ENGINE
'django.contrib.sessions.backends.db'
CSRF_FAILURE_VIEW
'django.views.csrf.csrf_failure'
CSRF_COOKIE_PATH
'/'
LOGIN_REDIRECT_URL
'/accounts/profile/'
LOGGING
{'disable_existing_loggers': False,   'filters': {'require_debug_false': {'()': 'django.utils.log.RequireDebugFalse'}},   'handlers': {'mail_admins': {'class': 'django.utils.log.AdminEmailHandler',                                'filters': ['require_debug_false'],                                'level': 'ERROR'}},   'loggers': {'django.request': {'handlers': ['mail_admins'],                                  'level': 'ERROR',                                  'propagate': True}},   'version': 1}
IGNORABLE_404_URLS
()
LOCALE_PATHS
('C:\\Users\\sksdutra\\Desktop\\Projetos\\soulmasters\\locale',)
TEMPLATE_STRING_IF_INVALID
''
LOGOUT_URL
'/accounts/logout/'
EMAIL_USE_TLS
False
FIXTURE_DIRS
()
EMAIL_HOST
'localhost'
DATE_FORMAT
'N j, Y'
MEDIA_ROOT
''
DEFAULT_EXCEPTION_REPORTER_FILTER
'django.views.debug.SafeExceptionReporterFilter'
ADMINS
(('Sandro Dutra', 'hexodin@gmail.com'),   ('Israel Crisanto', 'israel.crisanto@gmail.com'))
FORMAT_MODULE_PATH
None
DEFAULT_FROM_EMAIL
'webmaster@localhost'
MEDIA_URL
''
DATETIME_FORMAT
'N j, Y, P'
TEMPLATE_DIRS
('C:\\Users\\sksdutra\\Desktop\\Projetos\\soulmasters\\templates',)
SITE_ID
1
DISALLOWED_USER_AGENTS
()
ALLOWED_INCLUDE_ROOTS
()
DECIMAL_SEPARATOR
'.'
SHORT_DATE_FORMAT
'm/d/Y'
TEST_RUNNER
'django.test.simple.DjangoTestSuiteRunner'
CACHE_MIDDLEWARE_KEY_PREFIX
u'********************'
TIME_ZONE
'America/Fortaleza'
FILE_UPLOAD_MAX_MEMORY_SIZE
2621440
EMAIL_BACKEND
'django.core.mail.backends.smtp.EmailBackend'
DEFAULT_TABLESPACE
''
TEMPLATE_CONTEXT_PROCESSORS
('django.contrib.auth.context_processors.auth',   'django.core.context_processors.debug',   'django.core.context_processors.i18n',   'django.core.context_processors.media',   'django.core.context_processors.static',   'django.core.context_processors.tz',   'django.contrib.messages.context_processors.messages')
SESSION_COOKIE_AGE
1209600
SETTINGS_MODULE
'soulmasters.settings'
USE_ETAGS
False
LANGUAGES_BIDI
('he', 'ar', 'fa')
FILE_UPLOAD_TEMP_DIR
None
INTERNAL_IPS
()
STATIC_URL
'/static/'
EMAIL_PORT
25
USE_TZ
True
SHORT_DATETIME_FORMAT
'm/d/Y P'
PASSWORD_HASHERS
u'********************'
ABSOLUTE_URL_OVERRIDES
{}
CACHE_MIDDLEWARE_SECONDS
600
DATETIME_INPUT_FORMATS
('%Y-%m-%d %H:%M:%S',   '%Y-%m-%d %H:%M:%S.%f',   '%Y-%m-%d %H:%M',   '%Y-%m-%d',   '%m/%d/%Y %H:%M:%S',   '%m/%d/%Y %H:%M:%S.%f',   '%m/%d/%Y %H:%M',   '%m/%d/%Y',   '%m/%d/%y %H:%M:%S',   '%m/%d/%y %H:%M:%S.%f',   '%m/%d/%y %H:%M',   '%m/%d/%y')
EMAIL_HOST_USER
''
PROFANITIES_LIST
u'********************'

You're seeing this error because you have DEBUG = True in your Django settings file. Change that to False, and Django will display a standard 500 page.     


2012/10/25 Tom Evans <tevans.uk@googlemail.com>
On Thu, Oct 25, 2012 at 2:31 PM, Sandro Dutra <hexodin@gmail.com> wrote:
> Hi, I'm having a little problems with translations...
>
> I've various {% trans "anything" %}, everything is correctly configured, I
> do the makemessages and compilemesages with sucess, and everything works
> correctly, but I've a problem, when I insert special characters in *.po
> files, I get DjangoUnicodeDecodeError:
>
> DjangoUnicodeDecodeError at /
>
> 'ascii' codec can't decode byte 0xc3 in position 14: ordinal not in
> range(128). You passed in <django.utils.functional.__proxy__ object at
> 0x02BBEEF0> (<class 'django.utils.functional.__proxy__'>)
>
> Anyone can point a solution?
> Thanks in advance.

I think you have put characters into your po file that are not in the
correct encoding. Can you describe how you are creating these "special
characters"?

Also, the full traceback may be illuminating instead of just the exception.

Cheers

Tom

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.



--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

No comments:

Post a Comment