[ art / civ / cult / cyb / diy / drg / feels / layer / lit / λ / q / r / sci / sec / tech / w / zzz ] archive provided by lainchan.jp

lainchan archive - /λ/ - 13

File: 1492029822151.png (90.3 KB, 300x300, 61SA0Wq1P1L.png)


Python 3 general.

Share resources, PDFs and books. Also discuss and share projects you have been working on.

Beginner Questions are also welcome.


I made a script that uses the Lainchan JSON API to keep track of new messages in specified threads. It sends me a mail, whenever new messages are posted in the threads I specified.

Feedback and Tips for improvement would be appreciated. The script can be founde here: https://git.fuwafuwa.moe/mananapr/python-scripts/src/master


It's readable, looks like a nice little piece of code to me. You might want to put the initialisation of the post count in a function, though.

Another tip: since you just have to do stuff every 10 minutes, it might be smarter to set up a cron job to rund your script instead of your constantly running script with time.sleep(10).

And for further reading it could be cool to look into aiohttp instead requests to get familiar with the async features of python3.5, not really useful for a small thing like this, but I think it's really fun to learn this stuff.


The server took too long to submit your post. Your post was probably still submitted. If it wasn't, we might be experiencing issues right now -- please try your post again later. Error information:

What the fuarrrk is this? I'm tryna upload PDF's


same here

The server took too long to submit your post. Your post was probably still submitted. If it wasn't, we might be experiencing issues right now -- please try your post again later. Error information:

{"readyState":4,"responseText":"<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx/1.10.3</center>\r\n</body>\r\n</html>\r\n","status":413,"statusText":"Request Entity Too Large"}


I tend to be more comfortable writing C++ code, but I can program in python. My biggest issue is the fact that you don't really know what type something is without looking at it for a while, whereas in C-languages you know for a fact that int a will always be an int... How do you get around this? Do you just get used to it?

Also, what programming environment do you lains use? I use emacs+bash. I can't seem to post pdfs atm, but once the servers are up and running, I'll put my fair share of knowledge


File: 1492059944851.png (6.6 MB, 200x200, Data Structures and Algorithms in Python.pdf)


File: 1492060096905.png (360.35 KB, 200x200, requests.pdf)

uploads werk again that's cool


File: 1492060195146.png (4.96 MB, 200x200, Python Algorithms.pdf)


File: 1492060330438.png (2.67 MB, 200x200, Python 3 Web Development - Beginner's Guide (2011).pdf)


How stupid is this mini sqlite wrapper? Any comments appreciated.


#!/usr/bin/env python3
import sqlite3
import os
from collections import OrderedDict

class Database:
""" bundle function for less soykafty database-interaction """
def __init__(self, db_path, setup_query=None):
self.db_path = db_path
if setup_query:
if os.path.isfile(db_path): os.remove(db_path)

def apply_query(self, query, payload=(), close_conn=True):
conn = sqlite3.connect(self.db_path)
c = conn.cursor()
c.execute(query, payload) #eval
conn.commit() #apply
if close_conn: c.close()
else: return c

def read(self, query, fetch_number=1, result_abstraction=None, payload=()): # retun dict must be ordered dict
c = self.apply_query(query, payload=payload, close_conn=False)
assert c, "Error in read"
data = c.fetchmany(fetch_number) # TODO: possible performance implications of fetchmany?

# if we don‘t just want to return a list of tuples, but a list of OrderedDicts representing the data
# (we use OrderedDicts to be able to map the postitoin of a tuple element to a key of a Dict in a convenient manner)
if type(result_abstraction) == OrderedDict and len(data) > 0:
assert len(data[0]) == len(result_abstraction), "wrong lenght of given result abstraction"
result = list()
for data_tuple in data:
current_result = result_abstraction.copy()
for i, key in enumerate(current_result):
current_result[key] = data_tuple[i]
return result

return data

""" example.png """
if __name__ == "__main__":
db = Database("yomoda.db", setup_query="CREATE table quotes (quote text, dankness integer)")
db.apply_query("INSERT INTO quotes VALUES (?, ?)", payload=("Good will come to those who wait", 9001))

# le memez implying that 2010 xD xD (kill me plz ;___;)
results = db.read("SELECT * FROM quotes WHERE dankness > 9000", result_abstraction=OrderedDict([
("quote", ""), ("dankness", 0)

for result in results:
print("The dankness-level of the quote '{}' is: {}".format(result["quote"], result["dankness"]))


File: 1492060604544-0.png (2.05 MB, 200x200, Packt Publishing Mastering Python Design Patterns (2015).pdf)

File: .png (843.69 KB, 200x200, 042fef111e2468537952b853c27e72586e0db99cc46397c24c07646692d2ffbbdeleted)

flood control I am angry


I thought of using cron initially, but switched to time.sleep() so that the script would also work on systems that don't have cron like Windows.
And thanks for recommending aiohttp. It looks cool

>My biggest issue is the fact that you don't really know what type something is without looking at it for a while, whereas in C-languages you know for a fact that int a will always be an int... How do you get around this? Do you just get used to it?

Strangely, I never have felt this way. Maybe you'll get used to it.

>Also, what programming environment do you lains use?

I use vim+bash in a tiling wm.



I'll dump a few more once the uploads get fixed again.


I started learning python3 with 'Automate the boring Stuff with Python' and got to learn some cool modules like BeautifulSoup and Selenium.
Now I want to get started with Machine Learning in Python. Can anyone point me to some tutorials or books?
I don't know where to start


File: 1492062810433.png (2.8 MB, 200x200, Packt Publishing Flask Framework Cookbook (2014).pdf)


I've been working recently using a lot of map, filter and reduce functions inside Python and everything gets messy so easily. Do you know of any built-in solution for this?

Several languages have solved this problem of composing functions by different aproaches. For example Rust has all those functions as methods for the Iterator trait then you can concatenate easily several maps/folds/filters. Elixir has the pipe operator. Do you believe that implementing a solution for this is worth it?


It's alright if you delete them. Do you know what's wrong with uploading? It works on and off and gives the error in >>35


Not yet, keep an eye on /q/ for updates concerning this.


File: 1492105189124.png (8.17 MB, 228x300, OReilly Introducing Python (2015).pdf)


Even though Python has map, filter, etc., I think the "Pythonic way" (I don't really like that phrase, personally) is to use list comprehension syntax, and the itertools and functools libraries.


More likely, to use for loops. Guido really doe snot like functional programming, hence why it's such a pain in Python.


File: 1492124632798.png (354.98 KB, 200x112, 1491982900_2017-04-12_1366x768_scrot.png)

I've been working on Driftwood 2D, a tiling game engine. After a long break I got back into it, and released a new version a few days ago. It's in Alpha and I've just started writing the User's Manual, so you can't expect too much from it, but if anyone wants to play around reading the test data will tell you how to get started. The engine is its own API; scripts from the game can call any public function in the engine. So, the source code is the API reference currently, each important function being already well documented.

* Infinite graphical and walkable layers
* Resource caching
* Virtual filesystem for game data and patches, supporting zip files
* Tile and sprite animations
* Sound effects and music
* Configurable input handling
* Timed callbacks
* Lightmaps
* Rudimentary database for saving data between plays
* Simple widgets with TTF font support
* Fully scriptable in Python 3

* SDL2
* SDL2_image
* SDL2_mixer
* SDL2_ttf
* Python >= 3.3.3
* Python jsonschema (https://pypi.python.org/pypi/jsonschema)
* Python PySDL2 (https://pypi.python.org/pypi/PySDL2/)

GitHub: https://github.com/seisatsu/Driftwood
Latest Release: https://github.com/seisatsu/Driftwood/releases/tag/Alpha-0.0.4a


>Feedback and Tips for improvement would be appreciated.

- The json fetch and load is textually repeated instead of being factored out into a function.
- Count is initialized to 0 then incremented for every item in a sequence, so check out the len [1] built-in.
- You aren't using the "__main__" [2] test to protect execution.
- You are printing "Started Watching" every time.
- There's no way to exit cleanly.
- You are looping over the items rather than the keys in the initialization loop, even though the old count is unused.
- You don't need end-of-line semicolons in Python.
- In the send_mail call urls[url] - diff is the value you have in c.
- In the send_mail body use string formatting [3] instead of concatenation in the url line to get the str calls for free.
- Use a ternary if to get "post(s)" exactly right.

[1] https://docs.python.org/3/library/functions.html#len
[2] https://docs.python.org/3/tutorial/modules.html#executing-modules-as-scripts
[3] https://docs.python.org/3/tutorial/inputoutput.html#fancier-output-formatting


Have you thought of making it into a fantasy console type thing like pico8 only Open-Source and running python?


I've thought of adding an in-game console where you can run Python code on the fly to interact with the API, but I didn't think of allowing an entire game to be developed in-game. It's an interesting idea, but the problem is that the map editor is a separate program so I couldn't integrate it unless I made an in-game map editor. It's a thought for the future.


Another thought though is that once we handle mouse input you could just implement an entire map editor in game scripts and run it on the base engine. That's how powerful the scripting is.


I have the physical copy of that book and currently I'm on chapter 8. Everything up untill chapter 6 was pretty well done, but after that it seems that it's going downhill. There are too many code errors and the explanations seems rather too vague.


Python is utterly uncomposable. Weird lambdas and stupid distinctions between what's a statement and what's an expression, Because Guido Said So.
Unfortunately there's no way around it that isn't a huge pain to implement, it's probably better to use Coconut or Hy.


wasted more time than i'm willing to admit trying to debug a program because i didn't know for loops in python are a lie


I made this abomination because Reddit's Atom feeds are too cucced for xscreensaver to parse

It functions both as a CGI script (?r=<subredditname>) or as a standalone executable (subreddit name as first arg)

(Lainchan chokes on muh tabs)


What do you mean with this exactly?


I think the lainon refers to the fact that loops have no special scope in python (unlike e.g. C or Java)


i = 12345
for i in range(43):

>>> print(i)


as opposed to:

int i = 12345;
for (int i = 0; i < 43; ++i) printf("hi dont optimize away kthxbye\n");

printf("%d", i); /* prints 12345 */


also list comprehensions behave differently in python3 and don't "leak" the variable



>I think the lainon refers to the fact that loops have no special scope in python (unlike e.g. C or Java)
That's wrong and you should stop giving advice if you don't know how python scoping works. This issue comes up every single time someone mentions it and every wannabe(you) gets it wrong 100% of the time.



Why wouldn't you explain why it's wrong then



>A scope defines the visibility of a name within a block. If a local variable is defined in a block, its scope includes that block.

>A block is a piece of Python program text that is executed as a unit. The following are blocks: a module, a function body, and a class definition. [...]
(*not* loops; except for list comprehensions since they "[...] are implemented using a function scope.")

>The following constructs bind names: [...], for loop header

>If a name is bound in a block, it is a local variable of that block

Serious question, I don't really get why you say that this is wrong since the provided examples also work as expected.
Additionally, like every StackOverflow answer on this would be incorrect then (e.g. http://stackoverflow.com/questions/6167923/block-scope-in-python) etc., so it would be kind to elaborate since, assuming you're right, it'd be a pretty common misconception. Thanks.


This script I wrote generates a table of all possible IPv4 subnets. It was a great exercise to improve my understanding of network addressing.

  #!/usr/bin/env python
import math

def subnet_info(network_bits):
max_hosts = pow(2, 32 - network_bits) - 2
dec_octets = []

full_octets = math.floor(network_bits / 8)
remaining_octets = 4 - full_octets

for _ in range(full_octets):

partial_octet = 0
for binary_digit in range(network_bits % 8):
partial_octet |= (128 >> binary_digit)
remaining_octets -= 1

for _ in range(remaining_octets):

subnet_mask = ".".join([str(octet) for octet in dec_octets])
binary = " ".join([format(octet, "08b") for octet in dec_octets])

return (subnet_mask, max_hosts, binary)

if __name__ == "__main__":
rows = [["SUBMASK", "BITS", "HOSTS", "BIN"]]
row_widths = [len(column) for column in rows[0]]

for network_bits in range(8, 31):
subnet_mask, max_hosts, binary = subnet_info(network_bits)

network_bits = str(network_bits)
max_hosts = str(max_hosts)

for i, value in enumerate([subnet_mask, network_bits, max_hosts, binary]):
if len(value) > row_widths[i]:
row_widths[i] = len(value)

rows.append([subnet_mask, network_bits, max_hosts, binary])

for row in rows:
for i, column in enumerate(row):
width = row_widths[i]

if i in [0, 3]:
print(column.ljust(width), end=" ")
print(column.rjust(width), end=" ")



>full_octets = math.floor(network_bits / 8)
Check out the // operator.

>for [...] dec_octets.append

Check out list * int and list.extend.

>partial_octet |= (128 >> binary_digit)

You don't need a loop. You can obtain x bits in the high position by shifting them up from the low position. You can obtain x bits in the low position by shifting 1 by x then subtracting 1.

>if len(value) > row_widths[i]:

Check out the max built-in.

>if i in [0, 3]:

In general, use sets rather than lists for membership testing, for that logarithmic feeling.


The format language will do this for you.
  >>> data = [("Name", "Cost"), ("apples", 40), ("grapes", 8), ("pineapples", 100)]
>>> c1 = max (len (x [0]) for x in data)
>>> c2 = max (len (str (x [1])) for x in data)
>>> fmt = "{:<{c1}} {:>{c2}}"
>>> print ('\n'.join (fmt.format (*x, c1=c1, c2=c2) for x in data))
Name Cost
apples 40
grapes 8
pineapples 100
>>> fmt = "{:>{c1}} {:<{c2}}"
>>> print ('\n'.join (fmt.format (*x, c1=c1, c2=c2) for x in data))
Name Cost
apples 40
grapes 8
pineapples 100

Move your main block into a main method and call main() in the "__main__" if. That way you can be imported into the interpreter without execution, and executed afterwards if desired by invoking main().


I came across this snippet in python and am absolutely mystified as to how it works.

Supposing that types is an array and x is a list of strings equal in length to types. Each entry in x corresponds to an entry in types and only one will be a non-empty string.
the next() call returns the index of the non-empty element in x, which is then used as an index for types.

  types[next(i for i, e in enumerate(x) if e)] 

As far as I can see next() takes an iterable object and optionally a default value to return if the end of iteration is reached. The arguments given here really don't make sense to me, and yet it works.


Rather than keeping this process running forever and checking every 10 minutes, set up a cron job to run every ten minutes and have your code run once.


annoyance: if you have a bytes object, you can't iterate through it with a for loop, because it gives you an int rather than a byte.


>The arguments given here really don't make sense to me, and yet it works.

I guess the confusing part is that it looks like there are two arguments but there is really just one argument, a generator expression. Maybe it becomes more clear without tuple unpacking:

  types[next(t[0] for t in enumerate(x) if t[1])] 


>The arguments given here really don't make sense to me

With square brackets you would have a list comprehension:
[i for i, e in enumerate(x) if e]
that would compute all values and return them in a list. You could then find the first one at index 0. With parentheses you have a generator comprehension:
(i for i, e in enumerate(x) if e)
which returns an iterator that will compute its values one by one when they are requested. With one next call you get the first value and don't compute the rest.

The final piece is that when a generator comprehension is the only argument in a function call you would have two sets of nested parens, and python allows you to omit one set. Why? Because the BaDFaiL said so.

Here's an example. Let's say patterns is a list of compiled regular expression patterns and input is a string to parse. Then you would do:
result = next ((m for m in map (lambda p: p.fullmatch (input), patterns) if m), None)
to stop checking at the first successful match.

Would you prefer iterating over a list to yield lists of length 1 rather than the elements themselves? Didn't think so.


Thank you so much. I had spun off in the wrong direction and had been bothered by it all day.
I had no idea about the lone generator being able to omit nested parens. I'm not sure saving two characters is worth making things less clear, but it probably makes no difference to people who are more experienced.


Thank you for taking the time to give me feedback, I've revised my program based on it: https://w1r3.net/RXrjQd.txt

>You don't need a loop. You can obtain x bits in the high position by shifting them up from the low position. You can obtain x bits in the low position by shifting 1 by x then subtracting 1.

I implemented this like so:
  partial_octet = 0xff & (0xff << (8 - network_bits % 8)) 
The AND with 0xff is necessary to ensure the number doesn't grow beyond eight bits, since shifting in Python won't make bits „fall off“.

I've also decoupled the data generation and presentation by splitting the logic in the main function into a print_table and a subnet_rows function.

The way you suggested to find the longest columns using max() rubs me the wrong way as it needs more iterations than the „less elegant“ solution. This isn't a big deal for such a small amount of data, though I would prefer the most efficient solution possible.


Ignore the superfluous for loop on line 13, I've replaced that with dec_octets = full_octets * [255].



>>> from ch1progs import *

>>> ToDate(11, 19, 1995)
>>> def td(m, d, y): return (m << 12) | (d << 7) | (y % 100)
>>> td(11,19,1995)

##### ToDate's definition in the source file
def ToDate(m, d, y): return (m << 12) | (d << 7) | (y % 100)

can someone explain this to me?


Pastebin the entire ch1progs file.


ToDate is a function that compacts a date into a number... maybe.

n << m shifts n bitwise "to the left" m times. So, a binary number like 0101 bitshifted left 3 times would be like 0101000.

n % m is modulo or the remainder for n divided by m.

n | m combines two numbers with bitwise OR. So, if either bit in the same spot of both numbers is set to 1 the corresponding bit in the result is 1, otherwise 0. For example, 0101 | 0011 is like this:

The idea of this is to combine multiple numbers together using a number. A simpler example of this would be to store two numbers under 8 together like this:
  def combine(a, b):
return (a << 4) | b
a is padded to give enough space for b, and then combined with b with OR.

Anyway it's weird because I don't know why the year is being taken mod 100. Maybe we know the year is in the 1990s? Shifting the date to the left 7 spots gives 2^7 possible years, which is only 128, but given that the year is being taken mod 100 it makes sense. The other paddings make sense, as 2 ^ (12 - 7) is 32 and there are at most 31 days in a month.


I should also mention that this is a low level way of representing dates and not for Python. Python has one of the best date/time libraries around in the standard library (datetime), so in a real application use that. But if such a library didn't exist, you'd want to make it more like this:
  def to_date(m, d, y):
return (m, d, y)
...or this:
  class Date(object):
def __init__(self, m, d, y):
self.m = m
self.d = d
self.y = y
...or this (which is what I'd do if I really had to)
  from collections import namedtuple

Date = namedtuple('Date', 'm, d, y')
The bitwise strategy of storing numbers is more commonly found in languages like C, but even then only when memory constraints are really tight (i.e., not in programs that'd be run even 10 or 20 years ago on PCs, much less today).

Also as a nitpick, functions in Python by convention start with an uncapitalized letter while classes are CamelCase.


Lain was asking about the 0 result, obviously.


ToDate isn't doing what you say is in the source, since its returning 0 for y=1995, which can't happen with the definition you give (because of the OR). Either that or ToDate is changed later in ch1progs, so please do >>671



>partial_octet = 0xff & (0xff << (8 - network_bits % 8))

This way is fine as well, the point was to get rid of the loop. My suggestion was:
((1 << x) - 1) << (8 - x)
which doesn't need a final masking because it generates exactly the needed bits.

>it needs more iterations

Traversing the data matrix in either row major or column major order produces the same number of data accesses. You also get the same number of length comparisons.

>lengths = [len(row[index]) for row in rows]

>column_widths[name] = max(lengths)
You don't need to store the lengths and compute the max afterwards. You can use a single pass [1]:
column_widths [name] = max (len (row [index]) for row in rows)
You can also do the entire length computation in one go:
lengths = [max (len (row [col]) for row in rows) for col in range (column_count)]

># O(2nc)?

Order of growth notation and analysis both ignore multiplicative constants.

>column_widths = {}

Dictionary lookup by key is much slower than list sccess by index. You only need the dictionary for the format call, so you can compute the lengths in a plain list, by column index, and create the dictionary afterwards. This should be a dictionary comprehension [2] combining the length and label lists using the zip [3] built-in.

[1] https://docs.python.org/3/reference/expressions.html#generator-expressions
[2] https://docs.python.org/3/reference/expressions.html#dictionary-displays
[3] https://docs.python.org/3/library/functions.html#zip